Assessing
Quality in Research
Seminar
Notes – September 29, 1993
Written
by Ted Baker at Florida State University
Overview:
What is research?
- Advancing
the state of knowledge – What this means depends on the level:
- High
school = your own knowledge (e.g. via library search)
- In
industry = your company’s knowledge
- Academe
= the world’s knowledge
- How
do you know what is the state of the world’s knowledge?
- Publication
also address quality through referee system
What
is quality of research? General criteria:
- Correctness
– see also specific paradigms
- Novelty/originality/creativity
– opening up new possibilities for research or “surprise” value
- Relevance
to real-life issue of interest: economic or social importance
- Difficulty,
relevance to other known hard problems
- Foundational
value (usefulness in further developments)
- Cumulative
value (building on existing work)
- Impact
on other research and on practical computing
- Popularity/acceptance/image/style
– social acceptance among peers, Hopefully based on other criteria
Some
common paradigms in CS
- Theoretical
– define abstraction, prove properties (theorems)
- e.g.
automata theory, formal languages, computability theory, complexity theory,
queuing theory
- results
should be mathematically valid
- abstraction
should relate to actual computing problem of interest
- results
should increase understanding of problem
- best
if opens new area of research or if it’s a new proof/analysis technique
that has other applications
- experimental-quantitative
- build
a system, measure performance (in actual usage, or random data)
- quality
based on soundness (validation of experiment, control, correct
statistics)
- relevance:
problem addressed must be well defined, its practical importance must be
clear, and it must provide new information not already available from
theory or other experimental work
- should
have a supporting theory, for experiments to shed light on
- should
suggest ideas for more theory
- should
supplement theory, where theory is inadequate
- creative—invention
of new artifacts
- usefulness
- originality
- relationship
to previous work (demonstrable improvements) – expressive power, speed
and efficiency, ability to be extended, application of sound theory
- widespread
adoption
- synthetic—unification
or collection of other research
- improved
unity, clarity, ease of teaching
- originality
= nobody pulled it all together before
- cross-discipline
– look for new applications of computer science
- originality
relative to the application area
- soundness,
currency of computing theory and knowledge applied
- utility
in advancing the science or practice in the application area
- value
in opening up new theoretical or experimental problems
Hardest
part? Coming up with a problem that is –
- Hard
enough to be interesting
- Not
already solved
- Solvable
by you
- Not
being solved concurrently by lots of other people
Where
to look for problems
- Other
peoples’ research papers
- Stated
open problems, maybe not a good bet unless you have special background
the author does not
- Unstated
problems: oversights, variations, generalizations, combinations
- Practical
problems: applications