Toby Wise Computational psychiatry | Anxiety and depression Investigating neural & computational mechanisms underlying symptoms of anxiety and depression

New paper on attention during aversive learning

I recently had a new paper published, which is the first project I’ve completed as part of my current fellowship. In this blog post I mainly want to talk about how the whole thing came to exist, along with the challenges I faced along the way. I always find “behind the paper stories interesting, and it’s always reassuring to hear about difficulties others have faces and how they’ve overcome them, rather than simply seeing the final, polished project.

The study

First I’ll summarise the paper. I don’t want to go into too much detail here – you can read the actual manuscript if you’re desperate for details!

Rationale

I’m primarily interested in how dysfunction in aversive learning processes might be involved in anxiety disorders. One repeated finding in anxiety research is that anxious individuals tend to pay more attention to potential threats (e.g. negative facial expressions), but we don’t really know why. My hypothesis was that this could be due to the way people learn about the aversive value of these stimuli (i.e. how unpleasant they are).

What we did

To answer this question we used a relatively simple aversive learning task. Subjects had to estimate the probability of receiving a shock from each of two shape stimuli shown on screen concurrently, and we used computational models to look at how people learned these probabilities, and how they estimated uncertainty around their predictions. We used eye tracking as an index of visual attention while subjects completed this task.

The task

What we found

Subjects paid most attention to stimuli with high aversive value (i.e. those most likely to lead to a shock), but attention was not influenced by how uncertain they were about the shock probability. We also found that attention influenced learning – subjects learned faster about stimuli they paid more attention to. We also found a rather confusing result when looking at relationships with trait/state anxiety – more anxious people tended to learn faster about safety!

Amazingly exciting results

The story

Conception & design

I’ve been interested in attentional bias towards threat for years, so I’m glad to have had an opportunity to try to understand it a little more. I now mostly work on learning in anxiety, and to me it made sense to try to link aversive learning processes to one of the most studied features of pathological anxiety. This project was something I naïvely thought would be a fairly simple – just take an aversive learning task and add on some measure of visual attention, couldn’t be too difficult right?

My very first design idea, which was kind of terrible and would have likely ended in complete failure

My initial plan was to use a task where subjects learned about two stimuli simultaneously and add in a few dot probe trials to get a reaction time based measure of attentional bias (subjects should react faster to dots shown in the position of the most valuable stimulus). However after a bit of reading I came to realise that the dot probe has a few issues, particularly regarding reliability, and eye tracking seemed like a better idea.

Next came the piloting, which revealed a number of problems that required dealing with:

  • Binary choices vs. continuous probability estimates? I’d originally planned to go with binary choices but discovered that modelling worked a lot better with continuous shock probability estimates.
  • Timings – how long to show each phase of the trial for? How long to give subjects to make a response? Should timings be jittered to make them less predictable?
  • Positions of stimuli – did I want stimulus positions to be random or fixed?
  • Task duration – we obviously need as much data as possible but attention tends to lapse pretty quickly, and we can’t shock people forever!
  • Instructions – how much can we tell people about what they are supposed to do? A particular problem I encountered (and which I couldn’t really solve without giving people explicit instructions about how to work out the shock probability) was that a number of people used a gambler’s fallacy-like strategy – i.e. if I got shocked last time I’m unlikely to get shocked this time.

Behaviour proved the most difficult to optimise. I wanted to look at how value and uncertainty influenced attention, so I needed a task that required subjects to estimate uncertainty and behave in accordance with this. I also needed models that incorporated estimations of uncertainty (while not being too complex and having recoverable parameters). This required a lengthy iterative process of adjusting the task design and modelling approaches, which was hindered slightly by the fact that I had no idea what I was doing when it came to computational modelling (thankfully my co-authors are far more experienced than I am).

I’m a fan of open science, and I wanted to preregister this study. This didn’t go quite as I’d planned, it ended up being a pretty rushed and vague preregistration and I didn’t write down hypotheses that I felt I hadn’t really thought about in enough detail (one of the main findings in the paper regards attention influencing learning, which I’d planned to look at but when I wrote the preregistration I hadn’t get thought through exactly how I’d do this and so left it out - in future I’ll be more organised). The main reason for doing this however was exclusions – I knew from piloting that I’d have to exclude a few participants and wanted to have these criteria set in stone before beginning. Overall I am incredibly happy I preregistered the study. I can’t count the number of times I went back to the preregistration to remind myself of exactly what I was supposed to be doing!

Analysis

Analysis also presented significant challenges - I ended up using a number of methods I had little prior experience with. Most of the analysis relies on hierarchical Bayesian models, which I felt provided the best way of dealing with the data but which I didn’t know much about at all. Thankfully tools like PyMC3 made this easier than it could have been, but it was still a challenge.

However, as this was a learning experience I inevitably ended up making mistakes along the way which made it a slightly more traumatic experience than it might otherwise have been – and really emphasised to me how easy it can be to innocently generate false positives. My first analysis run produced some quite nice results – in particular, it seemed like more anxious people overestimated uncertainty, which wasn’t too surprising a finding. However, when I came to check over my model fitting I realised that something had gone wrong (which was not obvious to me at all at first) and the model fits were all a little off. Rather sadly, fixing this made my exciting result disappear and it was instead replaced with one that made little sense to me!

At this point, I was quite happy with everything and wrote up a draft manuscript. I’d found that both aversive value and uncertainty influenced attention, which was quite exciting. As I wanted to embrace open science in this project I had decided I wanted to make all my code and outputs available online, and this obviously creates additional pressure to make sure all the code works properly and is (relatively) readable, so I looked over my code thoroughly to check everything in detail. This is a rather code-heavy project, and it’s almost inevitable in this kind of work that bugs will creep in.

This is where I identified a bizarre bug in one of my eye tracking functions that I hadn’t noticed previously. Again, fixing this led to more disappointment – attention was no longer influenced by uncertainty. This is another point at which my decision to follow open science practices has clearly benefited my work – if I’d not planned to make my code openly available I almost certainly would not have checked it in so much detail and probably would never have identified this bug, and as a result the conclusions of the study would be quite different (and wrong!).

What next?

This was a big challenge, but I think resulted in a nice paper with some really interesting results. I learned a lot along the way, including skills that I’ve made good use of in subsequent projects.

One of my main aims is to tie dyfunction in aversive learning processes to symptoms of anxiety disorders. One of the most surprising results from this study is that more anxious people seemed to think that they were generally safer than less anxious people (this ended up tucked away in supplementary material in the paper but I think it’s still really interesting). This is something I’m currently following up on, but my initial work so far seems to suggest that this might not be a spurious result.


Paper: https://journals.plos.org/ploscompbiol/article?id=10.1371/journal.pcbi.1007341

Preregistration: https://osf.io/8rwcu/

Data: https://osf.io/b4e72/

Code: https://github.com/tobywise/attention-threat-learning

Running online experiments using Google Firebase

Online testing is becoming a popular way of running psychological experiments. Services such as Prolific allow large samples to be recruited and tested in a matter of hours.

However one aspect of this process that often has people confused is how exactly to host a task online and store data properly. Some services are available that make this straightforward, such as Gorilla, however many of us want flexibility beyond that provided by these services.

After trying many ways of running tasks online, I’ve settled on a method that works well for me, which uses Google’s Firebase services. I thought it might be helpful to share how exactly this works to make it easier for others to get these things set up, and to reduce the number of confused messages I’m receiving on Slack from Evan.

Introduction to Firebase

Firebase is a suite of database, authentication, and hosting services provided by Google. It’s primarily designed for mobile/web apps, but its ease of use and range of unified features make it a great way to host tasks online and deal with data that they generate.

Firebase is GDPR-compliant and provides multiple data storage locations, along with ways to restrict access to the database to ensure that it’s not accessed by people who shouldn’t have access.

Firebase is a paid service, but is free up to a certain point (dictated by daily/weekly usage) - for most tasks it’s likely to cost nothing.

There are two services provided by Firebase that are useful for us:

  • Database
  • Hosting

I will go through how to set up each of these in turn.

Setting up Firebase

The first thing you’ll need to do is open a Google account, if you haven’t already got one. You can then go to https://firebase.google.com/ where you’ll be given the option to create a project.

You’ll then be taken through a couple of steps:

Name your project

Choose a name for the project you’ll be working on - you have have multiple projects.

Google analytics

We don’t need any of these services, so you can say no to this.

Once this is done you’ll be able to open your project!

Setting up the database

Now, you can click on the develop option on the menu on the left of the screen to access the various tools you’ll need. First, choose the database option.

Next, click on create database. You’ll be asked to choose your security rules. For now, choose Start in test mode. We’ll be changing these rules later anyway, and this will make it simpler for you to to test things.

Then, choose your data storage location. This will depend on your circumstances, but if you’re dealing with GDPR in any way it’s best to choose Europe.

After a minute or so, you’ll have a database set up.

Note: Firebase provides two types of databases, Cloud Firestore and Realtime database. Make sure you’re using Cloud Firestore.

How the database works

Firebase uses noSQL style databases. This means that they’re not set up in a rows & columns format, and instead use a hierarchical structue consisting of what firebase calls collections and documents.

A collection contains multiple documents, but no data itself.

A document can contain two things: fields and more collections.

Fields are where your data itself is stored.

The way I’ve typically structured my data is as follows:

I have a collection called tasks, with one document for each task I’m running. Within each document I have a fields representing information about the task, and then a collection called “subjects” that contains a separate document for each subject.

This subject document contains fields with their subject ID, date they completed the task, and any other relevant variables for the subject. The subject’s document also contains collections, for example a separate collection for each trial. The fields within these sub-collections will then contain their responses for that trial.

Security rules

If you click on the rules tab on the database section of the firebase website, you’ll see options for restricting access to the database. This is important for two reasons:

  1. Security - we don’t want people accessing the data if they shouldn’t.
  2. Data integrity - we need to ensure that we don’t overwrite or delete data when multiple subjects complete the task. Each subject should only modify their own data.

To achieve this, we set up a system where each subject completing the task is assigned an anonymous ID which identifies them within the database. Access to the database is then restricted based on this ID - data is only written to the database if the ID given in the write request matches the record that is trying to be modified.

This is what we use the rules section for. Below is what I then use in this section.

service cloud.firestore {
  match /databases/{database}/documents {
    match /tasks/new_task/subjects/{userId} {
      allow read, update, delete: if request.auth.uid == userId;
      allow create: if request.auth.uid != null;
    }
     match /tasks/new_task/subjects/{userId}/{document=**} {
      allow read, update, delete: if request.auth.uid == userId;
      allow create: if request.auth.uid != null;
    } 
  }
}

I’ll run through what this means:

match /databases/{database}/documents {

When the subject runs our task, the task will request to write to a specific database. Here, {database} is filled in with this database, so this line essentially means that when a request comes in, it will only proceed if it matches the database we want to write to.

match /tasks/new_task/subjects/{userId} {

This then proceeds to the collection of documents I have for each subject. Each subject needs a document in the subjects collection (e.g. /tasks/new_task/subjects/Subject123).

In this line, the {userID} is filled in by the write request that the task sends whenever it tries to write data (this will be in the javascript code for the task, and we’ll go over it below). This means it just looks for a document with the subject’s ID.

allow read, update, delete: if request.auth.uid == userId;

If it finds a document that matches the subject’s ID, this allows the request to read, update, and delete the data in this document.

allow create: if request.auth.uid != null;

If the subject ID doesn’t match an existing document, this line gives it permission to create that document.

Together, this means that it looks for a document with the subject’s ID, if it exists it’s allowed to modify this document, if it doesn’t exist it’s only allowed to create a new document with that ID.

match /tasks/new_task/subjects/{userId}/{document=**} {

This line essentially does the same thing, but with the data contained within this subject’s document. So if it finds a subject with an ID that matches that being submitted, it can modify collections/documents within that subject’s document.

The overall function of these rules is to allow each subject to create data if it doesn’t exist, but only to modify existing data if their ID matches the ID of the data they’re trying to modify.

Authentication

In order to use these database rules, we need to provide each subject with a unique ID that Firebase can understand. The easiest way to do this is using Firebase’s own authentication service. This is designed for logging people into apps etc, but works well for our purposes.

This is simple to set up - all we need to do is enable anonymous sign-ins. This means that we can give each subject an anonymous ID that will identify them within firebase.

To do this, go to the authentication tab on the left menu bar, then choose Sign-in method at the top. At the bottom of the list of sign-in providers, you’ll see Anonymous. Just enable this and we’re good to go.

Adding Firebase database to a task

Once the Firebase database is set up, we need to tell our HTML/Javascript task to write data to it.

Detailed instructions for this can be found in the Firebase documentation

HTML

First we need to add some things to our HTML file (e.g. index.html) to tell it where to load the firebase code from. Then, we need to initialise Firebase. This means giving information about your database so that the script knows where it’s supposed to find the database you’ve created.

Thankfully, this is straightforward and firebase basically gives you something to copy and paste.

To get this, first go to project settings.

Then in the Your apps section, click on the </> icon.

When asked to give your “app” a nickname, enter a name for the project. You don’t need to select Firebase hosting for now.

Once you’ve done this, you’ll be given a code snippet to copy and paste into your HTML file, which will look something like this:

<!-- The core Firebase JS SDK is always required and must be listed first -->
<script src="https://www.gstatic.com/firebasejs/6.4.0/firebase-app.js"></script>

<!-- TODO: Add SDKs for Firebase products that you want to use
     https://firebase.google.com/docs/web/setup#config-web-app -->
<script src="https://www.gstatic.com/firebasejs/6.4.0/firebase-firestore.js"></script>
<script src="https://www.gstatic.com/firebasejs/6.4.0/firebase-auth.js"></script>

<script>
  // Your web app's Firebase configuration
  var firebaseConfig = {
    apiKey: "dsfjdshu4wru3rwefufguds",
    authDomain: "project.firebaseapp.com",
    databaseURL: "https://project.firebaseio.com",
    projectId: "project",
    storageBucket: "project.appspot.com",
    messagingSenderId: "83438438237409123",
    appId: "4:43344:web:256g4rfgdt3r23e"
  };
  // Initialize Firebase
  firebase.initializeApp(firebaseConfig);
</script>

JavaScript

Next there are a few things we’ll need to add to any Javascript that we’re using.

Enable persistence

Normally, when we add data to our database, we send a request and the data is added immediately. If for some reason there is no internet connection (e.g. due to WiFi dropping) this will fail and we’ll lose the data. Thankfully Firebase has functions to prevent this, which will retain the data and try to send it when the connection returns. We need to enable this however, which can be done with the following code (taken from here):

firebase.firestore().enablePersistence()
  .catch(function(err) {
      if (err.code == 'failed-precondition') {
          // Multiple tabs open, persistence can only be enabled
          // in one tab at a a time.
      } else if (err.code == 'unimplemented') {
          // The current browser does not support all of the
          // features required to enable persistence
      }
  });

Create a user ID

As we covered earlier, each user needs a unique ID that identifies them in the database, and we create this using Firebase’s authentication service.

This creates a global variable uid that identifies the user, and can be used in database queries.

// Sign in
firebase.auth().signInAnonymously();

// User ID
var uid;

// When signed in, get the user ID
firebase.auth().onAuthStateChanged(function(user) {
  if (user) {
    uid = user.uid;
  }
});

Create a reference to the database

We need to create a variable that refers to the database to allow us to interact with it.

This is quite simple:

var db = firebase.firestore();

Creating data

In my database, I start by creating a document for this subject usind their user ID.

db.collection("tasks").doc('new_task').collection('subjects').doc(uid).set({
    subjectID: subjectID,  // this refers to the subject's ID from prolific
    date: new Date().toLocaleDateString(),
    time: new Date().toLocaleTimeString()
})

To break this down, db.collection("tasks").doc('new_task').collection('subjects').doc(uid) is a reference to the subject’s document within the subjects collection for this task. Using the .set() method here sets the data contained within the subject’s document to whatever is provided (in this case, it creates fields representing the subject ID, the date and time).

Note: .set() sets the contents of a chosen document to the values provided, removing any information that was there before

Updating data

If we want to update an existing field in a document at any point, we can use the .update() method. For example:

db.collection("tasks").doc('new_task').collection('subjects').doc(uid).update({
    total_points: number_of_points
});

This update a field in the subject’s document containing some number of points.

Note: .update() updates existing values in a document

The best way to add data as the subject completes your task will be dependent on the task itself, but it’s fairly easy to figure it out using a combination of .set() and .update().

Firebase hosting

Aside from the database, Firebase also provides an easy to use hosting service. There are plenty of alternatives to Firebase hosting, and you don’t have to use Firebase hosting to use its database services.

To get started with hosting, click on the the hosting section on the left menu bar, and then click Get started. This will take you through a few installation steps.

Installing hosting tools

Firebase hosting relies on some command line tools on your computer, which need to be installed through npm. Npm is a package manager that comes with Node.js (which can be downloaded here).

Once you’ve got npm installed, npm install -g firebase-tools should install all the Firebase tools. You can then click next on the Firebase hosting setup page (you can leave the box about adding the Javascript SDK unchecked).

Next, go to the directory containing your HTML/Javascript files. First, type

firebase login

This will log you into your Google account.

Then, we can initialise Firebase and deploy the web page. First, make sure you have a directory that’s organised properly. Essentially, you just need a top level directory that contains a directory called public containing your web page.

Once you have this, navigate to the top level directory and use the following command to set up the Firebase project.

firebase init

Then, when you’re ready to deploy, run

firebase deploy

And that’s it, your task will be deployed at the url given in the Hosting section of the Firebase dashboard.

Collect some data

That should be everything you need to get your task set up and running using Firebase, so you can test it out and start collecting data!

If anything here doesn’t work please let me know, it’s quite possible there are errors as this was a bit of a rushed job!

How to survive a fellowship interview

Fellowship interviews are generally terrifying affairs that few escape without some degree of psychological distress. But it’s totally worth it, honest.

Having been through this and survived, I thought it might be helpful to write down some of the advice I’ve been given when preparing and given to other prospective interviewees. I’ve only interviewed for Wellcome’s Sir Henry Wellcome fellowship, so this post is based on this, but all early-career interviews share a vaguely similar procedure so hopefully this advice should be at least partially applicable to other schemes too.

The setup

You’ll be interviewed by a panel of 15-20 people, including academics and a few administrative staff. All at the same time. In one room. This is obviously pretty scary. They all sit at U-shaped tables, while you sit at a separate table in front of the panel. Thankfully only two or three of these people will be asking you most of your questions (the questioning is opened up to any panel members, but they often don’t ask many, if any) – these will be the people on the panel who work in fields most closely related to yours.

The procedure

Before the interview, you’ll be welcomed by your grant manager who will try their utmost to distract you from the upcoming horror. Soon enough you will be led into the room and sat down at your small table in front of the panel. The chair will introduce themselves and the two panel members who will be leading your interview. You’ll attempt to look around and smile at the entire panel, while in my case nervously drinking almost all of the water provided for you. Some of the panel may seem entirely disinterested, choosing to look at laptops rather than you – this isn’t a bad sign, they’re likely just preparing for questioning the next candidate.

The chair will then ask you to update the panel on any progress since you submitted the application and to give the panel a summary of your proposed project. In some cases, you are asked to prepare slides for your summary, to be sent one or two weeks before your interview date. The slides should be very effective, simple and visually oriented – think of TED talks. You want panel members to watch you, not to start reading lots of text – you are the main thing they are assessing once you get in there, so make sure their attention is on you and don’t feel shy to be at the centre of the attention. You’ll then rattle you well-rehearsed presentation off at about twice your normal speaking pace.

The first of your interviewers will then spend about 10 minutes (I think, honestly it feels like an hour but unless clocks don’t work properly in the interview room it isn’t actually) asking you questions, before handing over to the second interviewer who will ask questions for roughly 10 more minutes. The panel members will be sitting quite far away from you, so it is OK to ask to repeat a question if you are not sure you heard well. Once they’ve finished asking questions, the chair will ask if any other panel members wish to ask any questions. After this you’ll be asked whether you wish to ask any questions (most people don’t). They will then thank you for your time and you’ll nervously stumble out of the room and run straight to the nearest place glass of wine (handy tip – the Wellcome Collection is next door to the Wellcome building and serves alcohol, or there’s the Euston Tap across the road if you can make it that far).

You’ll then endure what feels like a year of worrying about how you missed a word out of one of your answers and having a recurring nightmare that involves being relentlessly quizzed by a panel of twenty people, before finally being informed of the outcome. If it’s good news, this is usually by phone.

The questions

It’s impossible to predict exactly what you’ll be asked, but you can generally guess some of the major ones. These will be things that you’ve identified as potential issues yourself, or that you weren’t entirely sure about when writing the proposal. If you’re given reviewers’ comments beforehand, this will help you identify potential weaknesses, but don’t rely on them – the questions I was asked weren’t too closely related to the reviews. Keep an open mind when preparing though and prepare for anything – both ‘bigger picture’ questions and more technical questions on your methods and hypotheses. The chair of the panel usually may ask a few more general questions on the strengths/weaknesses of the project or training plan.

The questions will mostly focus on the project itself, but it’s possible that you’ll get questions about what makes you a good candidate and why your proposed host institution is appropriate. Remember to focus on the 3 ‘P’s, you need to justify the person (you!), the project, and the place (including your sponsors). This inevitably risks putting you in the awkward position (at least if you’re British) of having to explain why you’re the greatest researcher to ever have graced the interview room, so get used to ignoring your self-doubt and focusing on your achievements – you wouldn’t be at the interview if you didn’t have a few.

The panel members tend to be friendly and genuinely interested in what you’re proposing and won’t be trying to catch you out. I have to admit that I found my interview fairly stressful – I got some unexpected questions that I wasn’t sure how to answer, and one of my interviewers was quite hard with their questioning. However, I expect I was being pushed; I’ve heard from previous applicants and panel members that sometimes they will attempt to push you to the edge of your knowledge, so don’t necessarily expect an easy ride.

When answering questions, remember to be concise. Nobody wants to hear you waffle for 5 minutes, so try to answer the question as clearly and succinctly as possible. Don’t try to be overly defensive in your answers; you don’t want to try to argue with the panel. This isn’t to say you shouldn’t defend yourself against critique, but it should be along the lines of “that is a valid point, but here is how I will address the issue” rather than “this is not a problem and you clearly have no idea what you’re talking about”.

How to prepare

  • Have mock interviews – this is the most important thing you can do. Mocks will get you used to being interviewed, make you aware of weaknesses in your project and your interviewing skills, and give you a rough idea of some of the questions you might get asked. It’s good to try to get interviews with more senior academics (the kind of people who will be on the panel), but I personally found mocks with my peers equally helpful. You will come out of every one of them feeling completely destroyed, but they are honestly the best prep you can do. Essentially, the more mocks the better.
  • Prepare answers to questions – mocks will help you get an idea of questions you might be asked, but it’s good to really spend time thinking about any weaknesses in your application and coming up with questions you think you may get related to these. Once you have an idea of what you might be asked, you can think about how best to answer these. I ended up with a ~7500 word document full of potential questions and answers, as I’d tried to cover everything I could possibly think of. You don’t need to memorise answers to 100 questions, but the practice of thinking about them will mean you get to know your proposal and its weaknesses intimately, making it easier to think on your feet on the day. I would however recommend trying to roughly memorise (not word for word) answers to any questions you’re fairly certain you will get asked.
  • Talk to previous interviewees/interviewers – if you’re lucky enough to know people who’ve either interviewed for fellowships or even been on the panel, chat to them! Most people who’ve done this will be happy to help. You can generally find the names of people that were awarded the fellowship you’re going for on the funder’s website.
  • Do some research on the panel - it’s worth looking at who is on the panel beforehand (they are often, although not always, listed online) to get a feel for who might be questioning you and what they might want to ask. For example if there’s someone on the panel who is an expert in the methods you’re planning to use, you might get some questions about your methods!
  • Rehearse your presentation – you need to know your three minute presentation off by heart. You want this to be clear and to the point, explaining exactly why you, the project, and the place you’re doing it are worth funding. Get as much feedback as you can on this, most people will be happy to listen to a three minute long speech! It’s also important to get feedback from people outside your immediate research area, as many of the panel members won’t have an intimate understanding of what you’re proposing.
  • Know your arsenal – Ultimately, any funding body just wants to make sure you will have the resources and support you need to carry out your work. Look in to courses you can take or people you can turn to (who may or may not have been listed on your application) for specific areas of your project – particularly those that you might not have much prior experience with. It’s handy to have these little tools in your arsenal during the interview to show the funders that you’ll be in good hands.
  • Read your application – this might sound obvious, but it’s surprisingly easy to lose sight of your original application amidst all your interview prepping. Do give your full application a once over before the interview, keeping in mind this is the only information (other than the reviews) your panel has to go on.

Hopefully this is somewhat helpful. The interview is a stressful situation and very few people come out feeling entirely happy, even if they end up being successful, but I think it’s a worthwhile experience regardless of the outcome – if you intend to stay in research (and you likely do if you’re applying for a fellowship), you’ll have to do this sort of thing again (and again) in your career so it’s worth getting some practice in early!

I’d like to thank fellow interview survivors Giorgia Michelini and Christina Carlisi for their help writing this post.

Making science with neural networks

As scientists, our primary aim is to publish papers in flashy journals such as Nature and Science (despite some claims that we should instead be focusing on extending human knowledge). In service of this aim, we spend immeasurable numbers of hours attempting to devise novel and exciting experiments that will seduce the editors of such journals and enable us to further our careers.

However, this seems a little inefficient. What if we could instead generate experiments that would be worthy of publication in these top journals instantly and automatically, leaving us to then simply carry out the work safe in the knowledge that the results would be gauranteed to appear in a top-tier publication?

I’ve recently seen many great (and hilarious) examples of neural networks being used to generate samples of text based on training datasets (http://lewisandquark.tumblr.com/ is worth spending several hours reading - she’s used neural networks to generate beer names, recipes, and name cats), and thought this seemed the ideal solution to this problem - can we generate ideas for papers with this new technology?

To try this out I retrieved titles of the most recent 50,000 papers published in Nature and Science, and set to work. I used a Python implementation of a char-rnn, based on Theano and Lasagne (I admit I did this half because I found the idea of writing import lasagne amusing), and trained it on these paper titles. After about 15 hours, it seemed to have a pretty good understanding of the entirity of science, and was producing some exciting, often strikingly inter-disciplinary, papers of its own.

  • The structure of the human gut microbiome in the solar system.
  • An extremely layer regulates biodiversity and the evolution of forest coupling at the centre of the global carbon cycle.
  • The genomic landscape of the electron pairing between the evolution of the southern ocean carbon nanotubes.
  • The genome of the supermassive black hole in an early mammals.
  • Antibody-mediated spindle of a single atoms in the tropical forest haematopoietic stem cell division in a compact density wave in the solar system.
  • Complexity of complex communities.
  • Structural basis for the control of protein activity in macaques.
  • Activation of the endoplasmic reticulum stress in the hippocampus.
  • A microbial genetic discovery of a protein phosphatase 2A and implications for the continental graphene transition.
  • A high-resolution structure of the Amazon deforestation and antibiotic resistance in the solar system.
  • A massive star formation in the abundant microbiota.
  • A progressive insight into the adult neural selection in a mouse model of the human genome.
  • Reconstructing the genome sequence of an extrasolar planet.
  • A resonant liquid from the leaving circulating wave revealed by self-renewing infective reactivity and temporal diversity in a stripped-resourcting the histone modification enhances neural responses to climate change.
  • A strong magnetic field in the hippocampus.
  • An atomic methanol using CO-methylatropospheric during the last deglaciation.
  • Antidepressants in a global clouds from modern human impacts on molecular channels.
  • Plant invasion by a radical proteome.
  • The missing memory in an exceptional patterns in metallic glasses.
  • A general model of the surface of the Southern Ocean deep states by means of interaction with neurotransmitter release.
  • A gene regulatory network states in a superconducting qubits via atomic clock on the contributor of a potassium channel Ca(2+) and PAL1 and toxicity in a tropical forests.
  • Discovery of a three-dimensional transition in the face of rapid warming leads to collapse of the Sun.
  • Complex carbon nanotubes in medial prefrontal cortex.
  • Complex structure of a metatherically stabilized reconstructed by massive mice.

It had even learnt that it was possible to write commentaries (it’s interesting to think about the thoughts such a commentary would contain given the subjects of these papers…):

  • Comment on “A common genetic variants associated with a single component of the mitochondrial calcium uniporter.
  • Comment on “A common genetic variants in the developmental disorder in the active site of the tropical Antarctic ice sheet.
  • Comment on “Density using mitochondrial fibroblastoma stemplling by the supersout restore responses in metal organic matter to ecosystems.

Some are even somewhat believable, if a little vague:

  • Control of the human transcriptome.
  • The structure of the human gut microbiome composition and its regulation.
  • Structure of the mammalian circadian clock.
  • Rapid early microscopic observation of inflammation.
  • Suppression of the southern ocean acidification.

It’s interesting that the neural network appears to believe certain phrases are particularly important to science today; genetic variants, structures, cirdadian clocks, and the southern ocean are frequently mentioned.

In summary, this has worked flawlessly and I will now be radically changing my research direction to focus on understanding complex carbon nanotubes in the medial prefrontal cortex. Expect to the see the results published in Nature soon.

How to apply for a postdoctoral fellowship in ten easy steps

Figuring out what to do after your PhD can be stressful. If you’ve not become disillusioned with the world of academia, you’ll probably be looking at a postdoctoral position. There are two real alternatives here: a job as a postdoc on someone else’s grant, or a fellowship.

Fellowships are an exciting option due to the freedom and opportunities they provide, however many simply aren’t aware of the available schemes or how to actually apply for one. This isn’t surprising given the complexities involved in applying for these positions, especially when you’re still relatively new to academia.

I was fortunate to do my PhD at an institution that actively helped early career researchers figure out their next steps and provided guidance on fellowship applications, but I’m probably in the minority here; many universities simply don’t offer this and I suspect that a lack of post-PhD support results in talented researchers being less successful than they otherwise could be.

For this reason, I’ve attempted to put together everything I’ve learnt about applying for a postdoctoral fellowship in the hope that it might be useful for others considering taking this route. This is based on what I’ve learnt from previous applicants, academics who have sat on panels and sponsored applications, and my own experience (I was recently fortunate enough to be awarded a Sir Henry Wellcome Fellowship from the Wellcome Trust, and my stress and confusion during the application process inspired this post). As I’m most familiar with Wellcome’s scheme for junior researchers, this is what I’ll focus on, however a lot of this will be common to fellowships from different funders and at different levels. This is also very UK-centric and based on my experiences in biomedical science (specifically neuroscience) - I’m not sure about options elsewhere but hopefully this information might still be useful!

Step 1. Decide whether the fellowship route is for you

Fellowships aren’t for everyone. The idea of a fellowship is that rather than working as a postdoc on someone else’s grant you get funded to carry out a project of your own design, working with people you choose to work with. For the funders, they’re an opportunity to support talented early career researchers, with the aim of propelling them to academic stardom. A strong fellowship application allows a talented person to work on an exciting project at a world-leading place (these three Ps are the three key ingredients of a good application).

Fellowship schemes are typically competitive, so you need to look pretty good on paper. As is unfortunately typical in academia this means having publications. Having one or two first author papers in “decent” journals should be sufficient here, assuming you’re around the end of your PhD. Evidence of having already been awarded funding (e.g. small grants for research or conferences) is also helpful.

Many people probably self-select out of the process because they don’t feel they’re good enough, even if this isn’t true. This is an easy trap to fall into, especially if you compare yourself with previous awardees who often seem to have superhuman levels of scientific talent. So be optimistic, it’s worth at least chatting to your supervisor to see if they think it’s worth applying.

It’s easy to assume you’re not competitive, don’t fall into the trap of comparing yourself to others as I did

Step 2. Think of a rough project idea

You need to propose a project that will lead to high quality research outputs, but which is also feasible in the time limit. At this stage you don’t need a detailed plan, just an idea of what you’d like to look focus on (for example, a couple of broad primary hypotheses you’d like to test). The project is one of the three Ps I mentioned previously - a good project is necessary for a strong application.

It’s important that this isn’t simply a direct continuation of your PhD work - there needs to be a clear training opportunity, for example learning new analysis methods, as fellowships are intended to foster your development as a researcher.

It’s fine to change field a bit, as long as you can show some continuity from your previous work, and at this stage it doesn’t matter whether your subject of interest aligns with funders’ priorities; this only comes into play at the higher levels, so don’t worry if your chosen subject isn’t “cool” right now.

Bearing in mind the time it takes to complete the whole application process, you probably want to have figured this out about 9 months before the deadline you’re aiming for.

Step 3. Find a fellowship scheme

There are a range of fellowships, which vary in terms of the career stage they’re aimed at and what they offer. Some are fairly generous in terms of duration and funding, while others are fairly brief or will cover just basic salary costs.

There’s sometimes a limit on how many years of postdoc experience is required to apply, even if this isn’t explicit (e.g. the MRC Career Development Award has no requirement, but you’re unlikely to be at the necessary level skills-wise without a few years of postdoc experience).

If you’re applying straight out of PhD, the most obvious choice is therefore the Wellcome Trust’s Sir Henry Wellcome fellowship, which is aimed at those of us at this level and doesn’t require any postdoc experience. There are two rounds per year, with deadlines around May and November, and they seem to award about 15-20 per round (you can view lists of previous awardees here).

Step 4. Identify potential sponsors and talk to them

A fairly crucial step is finding a sponsor(s) who will agree to host you. Most academics will be more than happy to chat with you about your ideas, so don’t be afraid to reach out to them!

Your host institution(s) and lab(s) should be the best place in the world to conduct the research, and you should aim to work with researchers who have a strong track record in the field - the place at which you choose to spend the fellowship is another important ingredient of the fellowship application, alongside the person and project. It’s very strongly encouraged to move away from where you did you PhD – staying within the same institution is likely to be criticised, unless this is clearly the best place for the work to be carried out (examples I’ve seen of this are where the candidate wishes to use a particular dataset or cohort which is based at the institute where they did their PhD).

These fellowships are also very flexible in terms of location, and this means you’re free to spend time at other locations (this was actually a requirement for my application). This is a great opportunity to broaden your horizons, gain new skills, and make contacts outside your host institution, so it’s worth speaking to multiple potential sponsors at this point about collaborations.

Speaking to potential sponsors will also help you refine your ideas for the proposal; they will be able to tell you whether your idea is feasible/actually worth looking at, and might point you in interesting new directions.

Step 5. Write the proposal

Writing a proposal is challenging, and a skill you will most likely not have much experience of at this early stage of your career. I’m not going to go into detail about how exactly this should be done as it could take up thousands of words, but I’ll provide some basic pointers.

It’s virtually impossible to outline a 4 year project in 1500 words. I spent a depressing number of hours rewriting and removing words to get it under the word limit. It’s also tricky to figure out how the proposal should be written – do you go for detail, or keep it more basic so that it makes sense to non-experts? Mine ended up being fairly detail-free, focusing on higher-level aims and hypotheses, but I’ve seen examples of both approaches that have been successful.

You’ll need to cover the background to the project, explain why your research question is important, and provide details of how you’re going to answer the question you’ve set. You’re not expected to have any pilot data, however make sure to reference any relevant work you’ve published already!

Your proposal should leave nothing to the reader’s imagination – make it obvious what question you’re asking and how you’re going to answer it. Make sure it’s clear how you’re going to spend your time, as it’s crucial that your project is obviously achievable in the timeframe – it can be helpful to include a timeline to illustrate this.

Try to contact previous applicants and ask to look at their applications – this is incredibly helpful when you’re figuring out how to structure your own.

Step 6. Rewrite the proposal

As with any piece of writing, getting feedback is invaluable. Your sponsors should be happy to look over the proposal – listen to their feedback, as they most likely have a great deal of experience in grant writing and have almost certainly reviewed grant applications themselves! Ask colleagues/supervisors/anyone who owes you a favour to take a look; Given how short the proposal is, it really needs to be as close to perfect as possible.

It’s also worth pointing out here that the proposal needs to represent your own ideas. Getting feedback from your sponsor is vital, but this shouldn’t be along the lines of “I don’t like your ideas, you should do X, Y, & Z instead”. Not only is this a pretty bad omen for your working relationship, it will also become apparent at interview that the proposal wasn’t your idea.

Getting feedback is crucial. However sometimes it turns out your friends are all too happy to be brutally honest

Step 7. Complete the application

The application requires various other details, such as statements of support from your supervisors/sponsors/mentor. Make sure you ask for these in plenty of time so you’re not left in a last minute panic!

You’ll also need to write a statement about your career to date and how the fellowship would further your career - this is how you address the third key part of the application, the person (you!). This is rather painful as you have to explain why you’re the greatest scientist ever to have lived, despite the fact that most of us seem to suffer from at least some degree of imposter syndrome. The key here is to use your achievements as evidence that you’d be successful if awarded the fellowship and make it clear how the fellowship would put you on the path towards an independent research career. Don’t worry if you’ve not received every award possible, published papers in Science/Nature, and cured cancer by the time you’re finishing your PhD – you might see people who’ve done this, but they’re the exception.

The application form will also ask for publications and awards. It’s fine to include submitted papers here (it’s fairly typical to not have all your PhD work published at this stage), but it’s not worth including work that’s “in preparation”.

Step 8. Submit the application

Eventually it’ll be time to undertake the terrifying task of submitting the application. Press the submit button and celebrate your hard work with a drink and sudden and terrifying realisation that you’ve not done any actual work for the past month.

Most anxiety-provoking button click of my life

Step 9. Submit the application again

The preliminary application will be reviewed and if you’re lucky you’ll be invited to submit a full application. Roughly half of the applications will be rejected at this stage.

The full application is largely the same as the preliminary application, with various extra bits of information (importantly, the proposal is largely the same at this stage so you don’t need to rewrite this).

You’ll need to speak to the people responsible for research grants at your host institution at this stage as they need to sign off on the submission, and they will probably expect you to do some sort of costing (nothing detailed, this is largely just a formality). Get this done sooner rather than later as you’ll typically only have about a month between hearing back and having to submit the full application.

After you submit the full application it will be sent to about three reviewers, who will provide feedback to the funder. The interview panel will judge whether or not to invite you to interview based on these reviews.

Step 10. Interview

If you’re lucky, you’ll be invited to interview. About half of the full applications will be selected for interview, so well done if you get this far!

The interview is where you’ll be quizzed on your proposal, and if you manage to wow the panel you’ll be awarded the fellowship! Roughly half the candidates interviewed will be offered the money, although this varies year on year.

The interview is worth a guide in itself, so I’ll leave this for now and hopefully address this in the future!

It’s probably pretty obvious now that the whole process is fairly involved. It’s a huge amount of work, and can be pretty stressful. There were multiple occasions where I felt like I had no idea what I was doing, and by the time I got to the interview I was adamant that I would never do this again. However, now I’m free from interview-related stress and panic, it’s obvious that it was 100% worth it.

Even if you’re unsuccessful, it’s a great way to get some experience of grant writing, which will serve you well in any academic career. It also enables you to build relationships with other researchers – if you don’t get awarded the fellowship, you’ve made strong links with your sponsors which could lead to other opportunities (note – these arguments seem entirely unconvincing when you’ve just come out of the interview feeling as though you’ve failed miserably).

I hope this has been helpful to anyone thinking of applying for a fellowship, and good luck!