Since May 2019 we have been participating in the IARPA 2nd Geopolitical Forecasting Challenge , Round 1 has just ended and we came 5th (out of about 40 participating organizations) overall and so have won a Round 1 prize!
IARPA is the US Intelligence Advanced Research Projects Activity (The Intelligence equivalent of DARPA) and its aim is to:
stimulate breakthroughs in the science of forecasting to create greater strategic advantages for maintaining global security, predicting economic trends and directing the need for humanitarian efforts.
The Challenge requires participants to forecast a large number of different events – political, financial, macroeconomic, technological, diseases, disasters, conflicts etc (for example – “Will the UK leave the EU by November the 1st”). There are 3 Rounds from May to November, with c 20 questions weekly. The wide variety of questions means having to handle a wide variety of data types, from time series (eg stock prices) to estimations using sporadic data, and some even mean resorting to game theory. Some questions end within days of being asked, others stretch for months ahead.
The 1st Geopolitical Forecasting Challenge was partly covered in the book Superforecasting. This 2nd Challenge is more challenging in three main ways:
- Larger volume of events to be forecast (c 300 in total), so requires use of algorithms rather than relying on ad-hoc analysis
- Forecast against all the problems set, not just picking and choosing those that suit your algorithms
- Heavier use of IT and API system building skills to interact with an automated IARPA challenge system.
For us the most telling scores in terms of validating our systems’ capability, are those showing how our systems are faring in the areas we designed them for. In Round 1 we scored 4th in Geopolitics (incl election prediction) and 10th in Macroeconomics. If we split out only the Financial prediction questions only (which our Financial system is optimized for), we came in 5th. So far no Technology and Business trend prediction question have come up so no useful results yet. (As an aside – if anyone thinks Geopolitics has little to do with Finance, just try predicting a few financial events in the near future).
It’s also challenging to have to adapt the DataSwarm system on-the-fly to answer all the question types where we have no built up data or tested algorithms – so an overall score “in the prizes” was also quite encouraging!
Anyway, we are hoping that we can carry on performing at this level over the next 2 rounds, but we were quite fortunate in Round 1, as quite a large % of the questions were financial and geopolitical, but in Round 2 so far the vast majority are of other types.
However, whatever the end results, by putting our systems (and us) through such a rigorous and challenging event we are learning a lot and also proving that the DataSwarm Analytics Engine works.
Incidentally, re: scoring – for each question your answer is scored for accuracy, and the average of all the scores of all the questions answered gives your overall score. The scoring system (Daily summed Brier scoring) uses the square of the size of the error in your forecast (brutal) and back-calculates for all the days from start to finish of the question – i.e. it penalizes you for how long it takes to arrive at an accurate estimate – so no pressure then…..