Category Archives: Teacher Responses to Student Writing

Teacher Response to Module #5: Hope We’re Still Friends ;-)

My peer review of your modules is complete and has been posted as individual feedback on Blackboard.

You do NOT have to make all the changes I suggested by the due date of the final project, nor should you try

Please remember that my goal here was to replicate the peer review process as much as possible. I thus responded like I respond to actual articles that come in for academic journals or conferences I am part of, which are the venues that all of you chose.

You might want to focus on 1-2 of my critiques and revise for those by the final due date for the project. DON’T TRY TO DO THEM ALL. It takes time to revise-and-resubmit, which is the response that all of you got.

Remember when I said that the write-up is the hardest part?

You have all conducted sound pilot studies. I would have told you by now if they weren’t sound, but writing about those studies is like a whole different skill set. You have to explain every in-and-out of your study and why you made the choices you did in a genre that you have never written in before.

Research articles and presentations are NOT the same as the papers you write in school, unfortunately. I think they should be, but this extra-curricular genre is so hard to write that I think a lot of faculty balk at trying to teach it. Really, we need a whole class in how to write these things. Not just one module in one class (or a few classes).

No one should even THINK about redoing your study

Don’t get discouraged by my peer reviews. No one gets write-ups right the first time, or probably even the 12th time, to be totally honest. And my reviews did focus on THE WRITING, not the research.

The way you describe your study, honestly, is more important than the way you conduct it. A lot of people would shoot me for saying that, but even if you do the best study in the world, if you never publish on it, no one will ever know about it, thus proving my point.

But you need to do an “actual” study before publishing, anyway

Please also remember that these are pilots, meaning that you CAN’T PUBLISH ON THEM IN ACADEMIC VENUES, YET. Some venues allow for the publication of pilot studies, but most don’t. If you’re not sure, write your venue’s editor and ask if they publish pilot studies.

The big hurdle is IRB: most academic publications require that research be conducted in consultation with an IRB.

That being said, IRB is just a submission away: http://www.ecu.edu/irb/. Well, the first thing you need to do is register for ePirate: http://bit.ly/1iCB7Cw. Here are directions for doing so if you’re unfamiliar with this system: http://myweb.ecu.edu/ohri/registration-ecuusers.mp4. When you do register, be sure you mark “I am a study coordinator or staff member of a research study team” as one of the roles you’re willing to fulfill.

Then you’ll need to complete your IRB training by going to http://www.citiprogram.org, creating an account, logging in, and completing the “Behavioral and social sciences investigators and key personnel basic course.”

Then, after all that, you can submit your study by going to ePirate: http://bit.ly/1s0pDwl. The just three short weeks later, voila! Yeah, it’s a lengthy, bureaucratic process.

Once you get IRB, you should collect new data

I often do pilots before conducting the “real” study, because they teach me a lot about that real study, which is often larger in scope. The real study is also an opportunity to get better data. What information did you need from participants that you failed to get? What are new kinds of participants you might recruit? Do you need to refine your research questions?

About the criteria I used for the peer reviews

The criteria I used are pretty standard for any academic venue. Let me explain a bit more about each criterion.

Significance/purpose: This is an assessment of what kind of contribution the research is for its specific venue. Is it new? Timely? Compelling? Does it build on existing scholarship, but contribute novel findings?

Methodology: This is an assessment of how sound the framework for your study is. Do the research questions match up with the theory, methods, and findings? Is there a clear connection between all these elements?

Findings: Are the findings articulated in sufficient detail that they are understandable, even to a well-educated non-specialist? Are they a direct response to the research questions? Are they significant? Is the “so what” of your findings clear?

Style: What about the language used in the study? Is it clear? Does it make as little use of jargon as possible? Are there concepts or passages that could be better explained?

Teacher Response to Module #4 and Homework #5: The Difference between a Research Report and a Research Article

Grades on Blackboard

Reporting vs. getting published

You’ve all created successful pilot studies and written up some very interesting findings. Unfortunately, these findings will never see the light of day if you don’t work 2x as hard to make them intelligible to a particular venue. Researchers read research articles and listen to research presentations. They don’t go beating down the doors of other researchers, outside of journals and conferences or meetups.

So, you’ve got to go to where your audience is, and you’ve got to learn how to sound like them. It’s as simple as that.

No one has original ideas

As some of you noticed, my recent article in Technical Communication was a direct imitation of the piece “Composing Across Multiple Media” that I assigned for Module #3. This is so taboo in the humanities that I hesitate even to point this out on our course website. It is the norm in the social sciences though, where methods and methodologies are considered communal property.

Why did I imitate Ranker so closely? It was an effective research design that worked for my study. Rather than reinvent the wheel, I took someone else’s wheel, measured it against my data, and said: hey, that fits pretty good!

Where a lot of training in the humanities gets it wrong is that we try to convince you to be “original” thinkers. This is like researching and writing in a vacuum, though. You’re going to create an “original” thought? Meaning something that’s never been thought before by anyone in the history of mankind?

More importantly: research doesn’t work that way. Research is a conversation. If you’re not willing to imitate, to take others’ ideas and test them out, then you’re talking to yourself.

#getsoffsoapbox

All this is to say: Try not to get upset when I respond to your drafts that are due on the 3rd

All this is building up to: I’m essentially going to give you a peer review of your drafts that you hand in to the website on the 4th. This means that I’ll be evaluating you based on the soundness of your research design and your write-up of that research design.

This will not be pretty. My goal will be to treat you as editors will treat you when you try to publish your first research article or present at your first major conference (WHICH ALL OF YOU ARE GOING TO DO, RIGHT?!). That is to say: I’ll be trying to find holes in your argument, gaps in your literature review, flaws in the way you’re describing your methodology, etc.

This is all part of doing research. Accepting peer critique and trying to improve your writing on your research is a necessary part of the process of becoming a researcher.

That being said: I will also include a “what I expect by the 11th” list, because you won’t be able to respond to all my feedback in just one week. The last time I got a revise and resubmit, it took me about a month to respond to all the changes, and being a researcher is essentially my full-time job.

Even though research is hard, writing about research is at least 2x as hard, in my personal experience. I’ve also been a serious researcher for about 10 years now, and I’m just now starting to get the hang of it. I don’t know if you ever completely get the hang of it though, honestly; you just learn how to roll with the punches better.

So, roll with the punches I send your way on the fourth and try to learn from my feedback. You all have done research that is way too good for it to never see the light of day!

Teacher Response to Homework #4

Grades on Blackboard.

Ethics Schmethics

When I talk to people in industry about the ways academics do research, they look at me as though I’ve just said “I love smallpox!” Industry researchers are all about solving problems and they don’t see why anyone would go through all the trouble to develop scripts, file IRB, etc.

Then I ask them what they’re doing and they say something like: “I’ve spent the last 9 months interviewing corporate executives about their workflow to help improve it.”

Me: “Did you reveal their identity to the public?”

Them: “No.”

Me: “Why not?”

Them: “They’re my client. That would be highly unethical.”

Me: “Exactly.”

My point is that everyone is invested in some form of research ethics, whether they realize it or not. Unless you’re a tabloid journalist, when you interview someone, you don’t want to defame them. Unless you’re a used car salesmen, when you survey a bunch of post customers, you don’t use that information to exploit them.

It is arguable that academics do things to extremes by overly bureaucratizing the process and requiring oversight on top of oversight on top of oversight. ECU is the second institution I’ve been at that has a “Committee on Committees,” whose job is make decisions about who’s making decisions.

What we do get right, though (at least most of the time, I hope), is research ethics, which should always protect participants, first, second, third, and forty-third. So, whenever you do a research project for the rest of your life, as yourself: is it in the best interests of your participants? Is there a potential to cause them harm? How can you mitigate that potential?

Teacher Response to Module #3

Grades + individual feedback on Blackboard.

Again, I’ll save you the trouble

My comments to everyone were pretty much “sounds good; now go collect some data!”

As I mention in the next Module, the most important things to consider during data collection are the following:

  • Doing what you said in your plan you were going to do. Even though this is a pilot, it’s important you try out the research design you’ve constructed.
  • Protecting your participants from harm. This means ensuring confidentiality, minimizing risk, and making sure they understand why you’re collecting information on them.
  • Not collecting too much data. Remember that this is a pilot. Your goal here is really to test out your research design, not to collect a ton of information that you’d then have to recollect to turn this into an actual study.

Have fun in the field! Let me know if you step in any bear traps!

Teacher Response to Module #2

Grades on Blackboard. The responses were pro forma…. see below.

I see research people*

You may have by now uncovered my secret plot behind this class, and that is to get you all so excited about empirical research that you can’t wait to do it for the rest of your life.

One of my firm life philosophies is that if everyone would do everything more empirically, meaning based on actual data they had collected themselves, then we could solve all the world’s problems. The opposite of empiricism, ideology based on opinion or doxa, is what creates and maintains many of the world’s problems.

I am overjoyed by the studies I am seeing and I can’t wait to see research instruments and eventually data. I hope you folks are as excited as I am by your studies, but I’m not sure that’s completely possible ;-).

Great work.

*In the tradition of “I see dead people meme

Teacher Response to Homework #4: Aaaaaah!!

Grades on Blackboard.

What is all this jargon?!

If you’re anything like me, at this point in the process of this class you may feel awash in a see of jargon. Attitudinal scales. Variables. Research instruments?!

It may sound like a different language. And guess what: it is. Those of us who have been researchers for a number of years have absorbed this very alien discourse and are “fluent” in it, so to speak. But it IS foreign, and it CAN be alienating.

Which is why research can be done poorly

Several of you have commented in your homework assignments that you’re starting to see flaws in published research based on what you’re learning in this class. Remember what our guest speaker said: it’s easy to do research wrong.

As you work to understand the concepts central to research design, please remember that people disagree on these concepts all the time. What makes for a valid study? How can a study be truly replicable? How extensive does a research instrument need to be?

Questions such as these have answers, but they are not universally-agreed upon answers.

Now you’re becoming part of the conversation

To find your way in this mess of terms and complex processes, please remember that the best way to learn any language is to jump in and use it. As you become part of this conversation, remember that you don’t have to agree with everyone else’s interpretation. That’s what fuels research, actually: collegial disagreement. So, if you think a term is poorly defined in our textbook, write it down for later study. If you’re not so sure attitudinal scales are valid means of analysis, look up critiques of them.

Be a researcher of research design, first and foremost. That’s the best way to become a good researcher.

Teacher Response to Module #1

Grades on Blackboard. Please also check for your individualized feedback there.

Some overall stuff to think about is below.

As Dr. Grabill said, research is hard to do well, and easy to do poorly

Almost everyone got their designs tweaked in one direction or another. And that’s okay. The best way to learn how to do research is to do it with someone with a bit more experience looking over your shoulder.

Your studies are evolving, and that’s okay, too

Many of you submitted documents that show that your thinking regarding what you are studying has shifted. Try not to get frustrated with that, but instead embrace changing your mind. Mostly what I am doing in this class, is pushing against your thinking to help you refine in and create sound studies. This process is, by necessity, messy and iterative.

Teacher Response to Homework #2

Grades on Blackboard, as per the norm.

Below is some stuff to think about.

Variables often emerge after a pilot

The reason we’re doing pilots in this class, as opposed to full-blown studies, is that pilots, or pilot-like activities, are often essential to defining variables. It’s often hard, when designing a study plan, to figure out what exactly you’re measuring, and there are typically a lot of different variables floating around in your study that you could use. The choice often comes down to what you are most interested in, and what you want to zero in on.

Best practices are important, too

That being said, what you decide to zero in on, and how you decide to collect data, should come from best practices within your field. As you are learning, research design is a complex activity, so the best way to do it well is to look to the successful research designs of others. Don’t  try to invent in a vacuum, because if you do, chances are your design will not be as good as it could have been.

Here are just a few things to watch out for:

  • The specific way you deploy a particular method. What are best practices for deploying this particular method and how are you responding to those best practices?
  • The variables you end up choosing, as compared to the variables of other researchers. If you’re studying variables that have never been studied before, chances are you are actually studying concepts, and have yet to define actual variables. The best way to avoid this is to mine other studies for variables.
  • The theory you’re using and how it is used within your specific field. Theories are some of the most politicized elements of research design. They come in and out of fashion, are deployed differently in different fields, and are often deployed poorly. To avoid this, look to researchers studying similar things and what theories they’re using. Avoid taking a theory from a completely different context and jamming it into your study because it’s something you’re familiar with.

Looking forward to seeing the study plans!

Response to Homework #1: Research is really, really hard at the beginning, but gets easier, I promise

Grades will always be submitted to Blackboard. Check there for them.

I also like to do whole class responses, however. I only do whole class responses to homework, but I do give individualized responses to Modules.

Why research is hard at the beginning

At the beginning, you may feel like you don’t know what decisions to make. How do you narrow your scope? How do you pick good questions to answer? How do you choose appropriate methods?

These are all difficult questions to answer, so my advice to all researchers-in-training is: figure out how other people have answered these questions first. Think about it, you wouldn’t start out trying to be a physicist with no knowledge of what our physicists are doing, right?

The same holds for all forms of research. Research problems/questions are maintained by professional communities. And if you’re just replicating what other people have done, then you aren’t really doing research. So: figure out what other researchers are doing, and go from there.

Research is a series of systematic decisions that mostly get made at the beginning of a study

The other difficult thing with research is that if this is your first study on a given topic (or your first study, period), you have more choices to make than if it’s your twenty-first. And worse, the decisions you make at the beginning of a study lock you into choices later down the line.

There’s an answer for this, too, though: it’s called a pilot study. Pilot studies are studies that you conduct to investigate a study design to test its validity. It’s a “dry run” before the real study. You should conduct a pilot study if:

  • you’re new to research
  • you’re trying a research design that is new, experimental, or untested
  • you’re not sure if your research design will help you answer the questions you’re asking
  • you’re not sure that there is a legitimate sample of data out there of the kind you’re trying to collect

That’s what we’ll be doing in this class: the whole class is a pilot study. So, you literally can’t fail at your research design in this class, unless you just phone it in, of course, which I’m sure none of you plan to do.

Research is about solving problems

As you will get tired of hearing me say: research should always be about solving problems. When I see many researchers struggling with a design, sometimes (not always) it’s because the research questions are too big or too small.

What do I mean by that? Research questions that are “too small” are insignificant, meaning they represent no real benefit to the research conversation from whence they came, often because they’ve already been answered, or are too general to really be helpful.

On the other end of the spectrum are the “I want to cure cancer and world hunger in my first study” questions, in which a well-meaning researcher takes on advanced questions that are too difficult to answer in one study, or maybe even in ten studies.

“Ideal” research questions should be heavily informed by a particular body of literature (e.g. through a thorough literature review) and specific enough to be resolvable in the study you’re actually conducting.

Example from my current research project:

Q1 (too small): are users of the North Carolina Coastal Atlas able to perform key tasks?

Q2 (too big): how do existing models for user experience design (UX) apply to Geographic Information System technology?

Q3 (just right): how can existing models for user experience design (UX)–particularly usability testing, contextual inquiry, and persona development–help improve the usability of an online coastal mapping application?

Anyway, these are some common pitfalls, many of which are mentioned in the readings, that I just wanted to reiterate. Excited by what I’m seeing so far from you guys!