$cover


short links


On May 24th 2021 the official UK Crowd Forecasting Challenge, hosted by the epiforecast group at the London School of Hygiene & Tropical Medicine will begin. Predicting how Covid-19 may spread is an important but difficult task - and we need your help!

The UK is at a critical moment. We have a successful vaccination campaign underway. However, at the same time new variants are emerging and spreading, which means easing restrictions comes at a risk.The coming weeks may hold important lessons for other countries coming out of lockdown. Many of these things are hard to model and hard to predict… which is why your help will be invaluable!

Over the course of 12 weeks, we’ll collect and score your personal predictions (forecasts) of the number of Covid-19 infections (which have been reported) and the deaths resulting from Covid-19 in the United Kingdom. The most accurate forecasters will receive a cash prize of up to 100 GBP.

Forecasts created through this competition will contribute to an ongoing research project about the value that crowd (public) forecasts can bring to public policy. After they have been quality checked, a summary of all the forecasts will also be submitted to the European Forecast Hub, a project created by the European Centre for Disease Prevention and Control to collect Covid-19 predictions. Therefore, you have a chance to make a real world impact through your forecast by providing valuable insights to policy makers and public health officials through the European Forecast Hub.

The forecasting apps used for this competition have been developed by researchers at the London School of Hygiene & Tropical Medicine to improve Covid-19 forecasting and help public health officials to make better decisions. The apps are free to use and treat your data securely in accordance with the Privacy Policy.

You can learn more about the background of this project in the about section at the end of this document.

How to sign up and get started

Visit either

Then you just create an account. All you need to provide is a username and a password (and your email address, if you want to win a prize). Once you create a user account, you will be given a random ID which links all your forecasts to you. Read more about how we handle your data here. In addition there will be a public leaderboard. You can either decide to appear on the leaderboard with your username, or you will be assigned a random pseudonym.

You will be automatically entered into the competition once you submit your first forecast for the UK (in the app you can submit forecasts for up to 32 European countries, but only forecasts for the UK will be counted for this competition).

Competition period and when to make a forecast

The competition runs over the course of twelve weeks. Every week between Sunday 12pm UK Time and Monday 8pm UK time we will collect your weekly forecast.

Ideally you should submit a forecast every week, but you don’t have to. You can miss a week or join part-way through the 12 week challenge. If this happens, you will be placed in the middle of the leaderboard for the weeks that you have missed (This does not mean you get the average of all forecasts, but instead you get a score that ranks in the middle of all scores for that date).

The competition will run from Monday 24th May 2021 until Monday 16th August 2021. In order to join the contest from the beginning, you will need to submit your first forecast before 8pm on Monday 24th May.

What will I be forecasting?

You will be asked to predict (forecast) both:

  • the number of Covid-19 infections and
  • the deaths resulting from Covid-19 in the United Kingdom

which you think will occur in the next 1-4 weeks..

Covid-19 infections in the sense of this competition means the number of of people who test positive for Covid-19, as reported every week by the UK Government. The number of deaths means the number of people who died within 28 days of receiving a positive test result for Covid-19, as reported by the UK Government.

Forecast submission

There are two ways you can make a prediction: You can use the ‘classical’ forecasting app or the new Rt forecasting app (more on these two below). If you use both app to make a forecast in one week, we will count these as separate forecasts, so in order to maximise your chances of winning, we highly encourage you to make a forecast using both apps! Forecasts from the two apps are independently eligible for a prize. So if you make a forecast using both apps, you have two chances at winning instead of only one (more on how exactly we score the predictions below).

Within the weekly forecasting period from Sunday 12pm to Monday 8pm UK time you can update and resubmit your forecasts as often as you like. For every app, we will only count the latest forecast (closest to the 8pm deadline). If you, for example, made 2 submissions with the classical app and 3 submissions with the Rt forecasting app on May 16 2021, then we will take the latest classical app submission as well as the latest Rt app submission and score these.

What could I win?

  • 1st prize: 100 GPB
  • 2nd prize: 50 GBP
  • 3rd prize: 25 GBP

After the competition we’ll contact you to about how to pay the prize money to you.

Eligibility

You need to submit at least one forecast for both measures (number of Covid-19 infections and number of deaths resulting from Covid-19 in the UK) to be eligible to win a prize.

In order to receive a prize, you must provide your contact details (either by providing your email address in the app user account, or by sending a message to epiforecasts [at] gmail.com if you don’t want to register your email address on the app. See more on how we use your data here.

While you may make two forecasts each week (one with either app) to increase your chances, you are allowed only one account.

You can resubmit and correct your weekly forecast as often as you like. For each app, we will only count the most recent eligible forecast, i.e. the latest forecast that still meets the Monday 8pm deadline.

The forecasting apps to use

There are two different forecast apps that work slightly differently. With the ‘classical’ forecasting app you are asked to predict future Covid-19 infections and the number of deaths resulting from Covid-19 directly. With the Rt forecasting app, you will be asked to predict the reproduction number R, which will then be converted into a forecast of Covid-19 infections and deaths for you. Here are the details:

The ‘classical’ forecast app

Link: app.crowdforecastr.org

Using this app, you will predict Covid-19 cases and deaths directly. The prediction has two components: the “median” prediction and the “uncertainty” around the median. The “median” is the value where you believe that there is exactly a 50:50 chance that the true value will be either higher or lower than that point. You should simply treat it as your best estimate for what is going to happen. You can specify the median value either by typing in a value, or by dragging around the corresponding points on the app.

The “uncertainty” captures how certain you are of your forecast. In the app, it is represented through a value called ‘width’. Greater width means more uncertainty. The exact numerical interpretation of the width value is not very helpful nor intuitive. We therefore recommend looking at the forecast and the uncertainty around it (which is marked in green) to decide what value is appropriate. One thing to make sure of, is that uncertainty increases over time.

Whenever you change any of the values in the app you need to press the ‘Update’ button in the app to reload the visual representation of the forecast. If you’re satisfied with your forecast, you can press ‘Submit.
Here are some more detailed instructions on how to use the app including a short video demonstration.

The Rt forecast app

Link: rt-app.crowdforecastr.org

Using this app, you will predict R (the average number of people that every infected person will infect in turn). This is the R value that you’ve heard a lot about in the media in recent months (in the app name it is called Rt to make clear that R varies over time t).

Predicting R works similarly to making a forecast with the classical forecasting app: you specify a median value and your uncertainty around it, then press ‘Update’. Your predicted R values will automatically be converted to Covid-19 infection and deaths using a so-called ‘renewal equation’ common in epidemiology (the study of diseases). You can read more about the very technical details of R and the renewal equation here. When you press ‘Simulate’ you can visualise the number of reported infections that would result from your current R forecast. However, the app won’t show you the corresponding forecast for the number of deaths resulting from Covid-19 so you’ll have to trust the renewal equation to produce sensible results.

There is some inherent uncertainty in the estimation of R even for values in the past. This is because it takes time for infections to be recorded and reported. We  therefore cannot know exactly how many infections there were at a certain point in the recent past and how many people those infected people have infected. In the app, you will therefore be asked to ‘predict’ values not only in the future, but also two weeks in the past. A dashed vertical line shows you the current date.

Remember, using both apps will always be an advantage for you as you have two chances at winning a prize!

More information about the different forecasting apps and a short instructional video can be found here.

What information can I use to make a forecast?

Any information (data) you like! Forecasts will be marked by comparing them to the “ground truth data” used by the European Forecast Hub. Even though different data sources may differ slightly, the ground truth data is the data the European Forecast Hub team has decided to treat as ‘the truth’. Currently, this ground truth data on numbers of COVID 19 infections and deaths is provided by the UK Government and presented by Johns Hopkins University in the United States.

Can I correct my forecasts?

Yes! Simply make a new prediction (forecast). The weekly forecast must be submitted between Sunday 12pm UK time and Monday 8pm UK time. Within that period from Sunday to Monday you can update and resubmit your forecasts as often as you like. We will only count the latest forecast (closest to the 8pm deadline).

I created multiple accounts. Is that ok?

We very much appreciate your interest in this project, but unfortunately you can only have one account per person.

What if I miss a week?

The way we mark and score your forecasts means regular participation during the 12 week period helps your score. But you can start submitting forecasts at any time during the 12 week competition or even miss a week.

If you are not able to submit a forecast before the 8pm Monday deadline on a given week, you will be placed in the middle of the leaderboard for the week(s) you have missed.

What if I forgot my password?

Simply write an email with your username to epiforecasts [at] gmail.com

How are forecasts marked?

We use a scoring system which guarantees that nobody can cheat and everyone is encouraged to provide their best possible forecast (prediction). We’ll combine your predictions for: 1. The number of Covid-19 infections and 2. The number of Covid-19 deaths to give you an overall score.

Your ranking will be based on the “weighted interval score” (WIS) of your forecasts. The ‘score’ is negatively oriented, meaning that a lower score is better. You can think of the weighted interval score as a ‘penalty’ for being less than perfect. The weighted interval score is the sum of three components (i.e. three different types of penalties): “overprediction”, “underprediction” and “sharpness”. Overprediction and underprediction are penalties that occur if the true observed value falls outside of the range of values deemed plausible by your forecast. If you make a very uncertain forecast, the range of plausible values is larger and you are less likely to get penalties for over- and underprediction. The “sharpness” term on the other hand penalises you for being overly uncertain. This means that you need to finely balance the line between being too uncertain (you get a sharpness penalty) and too confident (you get over- and underprediction penalties if your forecast is off).

To make forecasts of deaths and reported infections more comparable, we will first take the logarithm of your forecasts and the “ground truth data” and then calculate the weighted interval score using these.

If you miss a week and don’t provide a forecast on a given forecast date, you will be assigned the median score of all participants who submitted a forecast on that day. If you make a forecast using only one of the two apps, your score for the other app will be the same as the score for the app you used.

About the project

Forecasting epidemics - an ongoing research project

The UK Covid-19 Crowd Forecasting Challenge is part of an ongoing research project by the epiforecasts group at the Centre for the Mathematical Modelling of Infectious Diseases of the London School of Hygiene & Tropical Medicine in collaboration with Public Health England (PHE) and Imperial College, London.

Crowd (public) forecasts in the past were often able to beat model-based approaches when it comes to predicting the number of Covid-19 infections and deaths.

Over the past few months we have used our two forecast apps to collect crowd forecasts from a small group of participants, first in Germany and Poland and now across 32 countries in Europe. These forecasts have been submitted to the European Forecast Hub and the German and Polish Forecast Hub. In both countries, our crowd forecasts were among the top models (predictions) submitted (see this paper about the German and Polish forecasts or the evaluation of the European Forecast Hub forecasts). We therefore believe that crowd forecasts can be a valuable contribution to public policy and want to expand our approach to other members of the public.

While we know that crowd (public) forecasts can provide immense value, there are also many open research questions we want to answer.

  • What is the best way to combine individual forecasts together that will perform better than its individual parts?
  • How consistent are forecasters in their ability to forecast?

In particular, we want to look into a radically new way of forecasting. In addition to forecasting Covid-19 deaths and infections directly, forecasters will have the option to use the new Rt Crowd Forecast App. The app can be used to predict R, (the average number of people which a single infected individual will infect in turn). The R value gives an indication of whether Covid-19 infections will go up or down in the future. The predicted R path will then be mapped to Covid-19 infections and deaths using a ‘renewal equation’. We hope this will be a way to combine the strengths of human forecasters (predicting a trend) with the benefits of computer models (dealing with the complex specifics of epidemiological modelling).

We look forward to your participation!

Privacy

If you want to learn more about how we handle your data, take a look at our privacy policy.