Online Experiments for Language Scientists

Academic year 2025-2026

Undergraduate (LASC10115) Assignment brief, 2025/2026

Assessment 1 is an annotated bibliography where you review and evaluate papers; Assessment 2 is a more open-ended project where you produce a working experiment and accompanying report explaining what the experiment is for. Ideally the two pieces of assignment will fit together, i.e. you will read some stuff that appears in your Assessment 1 annotated bibliography that gives you ideas for something you could attempt for the Assessment 2 work. I understand this may not be the case though, and that your ideas might change or you might not be sure what you want to attempt for Assessment 2 until quite late, and you will not be penalised for this - the intention is to make it easier for you to think of these two assessments as related steps on a project, rather than two completely disconnected arbitrary things you have to do, but you are not forced to think of them in that way!

Assessment 1: annotated bibliography, due 6th November, worth 30% of course mark

Provide an annotated bibliography covering 4 articles. These articles can be drawn from the course set readings, but do not have to be (i.e. they can be other papers you have read); the only constraint is that they have to be relevant to using online experimental methods to study language or language-relevant phenomena. Ideally these will be articles which you found particularly interesting (for positive or negative reasons) and are likely to influence what you attempt for Assessment 2.

For each article in the annotated bibliography you should:

Provide the citation in APA format (as exemplified in e.g. the course web pages)
Briefly summarise its content
Provide your evaluation, e.g. features you found particularly strong or particularly weak, interesting relationship to other papers you have read, applicability of methods to other topics, applicability of a lab-based experimental method to online data collection, …
For articles which you think will be particularly relevant to your (current) plan for Assessment 2, highlight its relevance and how it has shaped your thinking.

A length of roughly 250-300 words per entry is appropriate; the maximum length for the entire bibliography is 1500 words. This is intended to be short, we are looking for concise and clear summaries and evaluations, so don’t strive to make it long-winded: the word limit is a limit not a target.

Additional notes:

When providing your summary, make sure this is in your own words - do not simply recapitulate the author’s summary or the paper abstract. Make sure your summary does the groundwork for your evaluation (point 3 above) and relevance for your Assessment 2 plan (point 4) - e.g. you will need to summarise those aspects of the paper that you touch on in answering those points.
Your evaluation should focus on the scientific content of the paper (e.g. strengths or weaknesses in arguments, methods, analysis, …), rather than superficial criticism of the writing style etc, unless that impinges on the science (e.g. a terrible literature review which misrepresents a theory, a misleading style of data presentation).
In marking the annotated bibliographies we will be looking for evidence that you have read the article (as indicated by an accurate summary in your own words), you have thought about it (as indicated by e.g. sensible evaluative comments) and that you have thought about how it relates to your own work or other work in the field (as indicated by sensible comments on connections to other papers or to your own plans for Assessment 2).
There are numerous examples of annotated bibliographies available online, including e.g. https://library.leeds.ac.uk/info/1401/academic_skills/80/annotated_bibliographies (but their example annotations are shorter than I am looking for) and https://owl.purdue.edu/owl/general_writing/common_writing_assignments/annotated_bibl iographies/annotated_bibliography_samples.html (more like the right length, maybe lacking in critical evaluation). Also see the FAQ for the assessment which includes examples of good annotated bibliographies for this course.

Assessment 2: coding project plus report, due 4th December, worth 70% of course mark

Assessment 2 has two parts:

A functioning experiment running on jspsychlearning.ppls.ed.ac.uk
A report explaining the motivation behind that experiment: i.e. the research question it is intended to answer (including briefly reviewing relevant literature and justifying that research question in terms of broader questions in the field), explanation/justification for any complex or questionable design decisions you took in designing the experiment (there’s no need to justify low-level stuff), an appraisal of weakness of your experiment or possible ways it could be improved/extended.

The experiment has to be relevant to using online experimental methods to study language or language-relevant phenomena, but beyond that there are no constraints on what you tackle - please have a short conversation with me (Kenny) if you are at all unsure about what you have in mind is appropriate.

We will assess these final projects based on two components: the technical ambition and implementation of the experiment, and the quality of the accompanying report and explanation. For technical ambition / implementation we will give high marks to challenging coding problems (e.g. going beyond the template experiments we provide for practicals), and experiments that work well and look good. For the report we will give high marks to projects that are well motivated by the literature, answer interesting well-explained research questions, and demonstrate interesting critical insights on your own work (e.g. interesting thoughts on design decisions or methodological weaknesses, even if you weren’t able to resolve those in the code).

You can choose how you weight your effort across these two components, e.g. if you tackle a demanding coding project we will be satisfied with a short report (e.g. 1000 words or less); if you are less ambitious on the technical side (e.g. largely re-using code we provide and ‘just’ plugging in different stimuli, trial lists etc) then we would expect a more ambitious report (e.g. longer, with a detailed literature review situating your work in the literature in a detailed way). The maximum length for the report is 2000 words. Again, please have a short conversation with Jenny if you want some guidance on how you should balance up these two components or how you should make sure you are doing something at the appropriate level of ambition.

Important points on providing a working URL for your experiment:

Please include the URL of your experiment (hosted on the jspsychlearning server) near the top of your report so we can access it. This will be of the form https://jspsychlearning.ppls.ed.ac.uk/~UUN/path where UUN is your student number (e.g. s24…) and path is the path to your experiment’s html file. If you include a link to the local files on your computer rather than a version of the experiment on the jspsychlearning server, we won’t be able to access your code and you will fail. If you include the wrong URL (e.g. using the “Open URL” option in cyberduck, which generates a URL with an error in it) then we may have difficulty accessing your experiment, and you may fail. We show you how to put experiments on the server and access them via a URL in the labs, so make sure you understand this!
We will run through everyone’s experiment, so if there is anything we need to know (e.g. if you have to give us a password to progress past a certain stage, if we need to run in two browsers simultaneously, etc) please include this info at the top of your report.

Additional notes:

This assignment is intentionally open-ended - I want everyone to engage with both aspects of the course (engaging with the literature, building actual web experiments), but I also want to be flexible enough to allow you to focus on what you are most interested in and/or what you feel will best illustrate your engagement with the course, and to allow flexibility in terms of how technically challenging your project has to be - not everyone is going to become a coding whizz in 1 semester, you are all starting from different levels of coding experience, but the intention is that this assignment is flexible enough to allow everyone to get a good mark.
Because it’s open-ended, we are relying on you to not overcommit on this one assignment - while nothing in the format prevents you from handing in a challenging programming exercise and a long detailed report, this is not necessary to get a good mark and you will eventually face diminishing returns.
It’s up to you whether you want to build an experiment with a novel element (e.g. a tweak to an existing experiment, or something entirely new) or one which closely replicates an existing study in the literature. Either is fine, but if you are focussing on replicating an existing experiment then you need to justify why that replication is necessary in your report (e.g. do you have reasons to be sceptical about the original result, is it particularly important that its robustness is checked, does going online allow you to access some theoretically-important new population, …).

Course main page

XXProject maintained by jennifer-culbertson Hosted on GitHub Pages — Theme by mattgraham