Your assessment question is going to drive the methodology you choose for data collection, and conducting a literature review is one of the best ways to gain insight on the benefits and drawbacks of different methods that have been used to answer similar questions.
Selecting a tool that is valid and reliable
If you intend to use or create a tool for data collection (survey, rubric, etc.) as part of your assessment project, you will need to consider whether that tool is valid and reliable. A tool is considered valid when it measures what it purports to measure. A reliable tool is one that gives the same value when the same thing is measured multiple times, including by different users.
Conducting assessment with a tool that is not valid and reliable may produce misleading results. Tools that are created internally need to be tested for their validity and reliability. It is often preferable to find a tool that has already been shown to be valid and reliable, and conducting a literature review is probably the best way to find such a tool.
Sampling considerations
Unless you plan to include the entire population of interest in your study (e.g. every student at your university or every book in your collection), you will need to select a sample. Some common types of sample selection are listed here.
- Random sample: Random sampling is the gold standard for research, but it is often difficult to achieve a truly random sample. In a random sample, each member of the population of interest for the assessment must be identified and have an equal chance of being selected. Using a random number generator is one way to achieve this, as is drawing names out of a hat. Even if your initial sample is randomly selected, there is a good chance that some of those selected will choose not to participate, which will make the sample nonrandom.
- Stratified random sample: A stratified random sample is representative of the population with regard to certain variables that have been deemed important. If you wanted to make sure that each college at your university was proportionally represented in a survey, you would divide the whole population into subgroups (colleges) and randomly sample a percentage of each subgroup for survey distribution.
- Purposive sample: In purposive sampling, the researchers handpick participants based on characteristics that would make them particularly good sources for the research at hand. For example, an assessment of the depth of the library collection in applied linguistics might specifically select faculty members who are known to do research in that area to interview them on their experiences.
- Convenience sample: Also known as accidental sampling, a convenience sample consists of volunteers. While very common, these samples have a bias in favor of participants who are more likely to volunteer (to take a survey, be in a focus group, etc.), and it will behoove the researcher to consider how this might affect their results.
Your sample size may be dictated by practical considerations such as time and money, but keep in mind that for quantitative assessment, your sample size may restrict the statistical analyses you are able to conduct with your results. For more information about sample sizes, see (link).
Human Subjects
If your assessment involves human subjects, you should consult the institutional review board at your campus to determine whether you will need to submit a human subjects application for review.
