Assessment 4

I have listed my contribution in relation to the following:

  1. Understanding of evaluation (theoretical and practical) in an individual and professional context – Personally I think that evaluation is a driver in my life as I do constantly think about making things better than they are. This normally means changing completely how I do things at present to have a better outcome or revising how I do things to make the process much easier and more efficient for myself and my family. Professionally I duly note the evaluation from my students and peers and consider how to improve my teaching and my interaction with my colleagues to have better outcomes.  To me Reeves and Hedberg’s (2003) discussion of the evaluation functions within the development process stand out as I can relate to it having designed and developed the iLessonPlan (iLP). I wish I had included this aspect when I had designed the iLP.  This is something that I’ll certainly consider when designing new programmes and courses and elearning tools in future. The Mixed Methods Evaluation model whereby both quantitative and qualitative data are used to provide rich data are extremely useful to make well informed decisions pertaining to the outcomes of formative and summative evaluation as well as needs analysis. This too is something I’ll certainly consider for future purposes.
  2. Evidence of learning and reflections about evaluation – including eLearning guidelines and quality in a relevant context – elearning principles such as ‘Learner centred’ , ‘Good practice’,  ‘Collaboration’, ‘Innovation’ and ‘Sustainability’ (E-Learning Guidelines, 2008; 7) provide useful guidelines in relation to evaluating the quality of elearning teaching and resources. In our evaluation report we identified under ‘Learner centred’ the usability of the iLessonPlan as a key point to investigate. Also under ‘Good practice’ we explored whether using the iLP enabled trainees to understand the key components and the structure and flow of the lesson better. These two elements were selected as they evaluated the iLP in relation to what it was designed to do. As the iLP is a prototype lesson planning took it was essential to evaluate these elements as the findings will be used to inform decisions pertaining to further development of the tool as next year a website will be created to host it with all its functions and functionalities.
  3. Reflection on the phases involved in the group evaluation project – Doing the evaluation plan enabled me to set my thoughts and ideas in concrete about how to go about the process even though I thought I had a pretty good idea of how to proceed. Also seeing it on paper enabled me to revise various parts of the plan so that was good. Data collection was a process where more time should have been allocated as there were several problems which cropped up (as reported in my earlier blogs) and sorting it chewed up time. Also in future I must make sure that I am more familiar using a tool for e.g. Google docs before using it for a project. The reason being that familiarising myself with the tool as well as using it at the same time also chewed up more time. The same goes for using Excel as my knowledge in that was more basic than was required for the project. The analysis to me was time consuming as I took some time to get my head around the data but once I had done that I was able to proceed. I was more comfortable reporting the results using tables but charts and graphs were suggested to be visually more appealing. This brought home the point that I needed to make the report reader friendly. Interpreting the results I thought was pretty straight forward but in future I will ensure that I know how to use the tools more effectively. I fully appreciated the feedback on the assessment drafts from my colleagues and Bronwyn. After discussion with Tom and Mark, suggestions deemed relevant and necessary were included. The process involved in completing the evaluation report is not linear but cyclical and at the end it was both at the same time. In future after completing any report, I’ll give it some time to breathe as it were and then complete the final proof reading.
  4. Personal contributions to the group process and working in a group – I have blogged about this in great detail in my other blogs and in summary I felt the need to lead and direct the team as I had designed the iLP which was being evaluated. Both Tom and Mark were excellent team members and we worked well together in pooling together our knowledge, experience and expertise in completing the assessments. I have learnt it is essential to respect each other’s views, and work constructively towards positive outcomes.
  5. Individual reflective learning about the process of evaluation – Evaluation is something as I said I do constantly in my quest to improve my knowledge, experience and expertise. I do focus consciously on doing things better be it my teaching, working with my colleagues or working on. assignments and projects. I also know I make mistakes and to me the important thing is what I learn from them, move on and learn how to do things differently in the future to have more positive and beneficial outcomes and outputs.
  6. Critique of the process used in one other group project and feedback provided at each stage – I gave feedback to the following groups:
  • Assessment 1 – Fred; Sandra H and Rosanne – I gave detailed feedback to both as seen in the Discussion Board.
  • Assessment 2 –  Sandra H and Rosanne’s Assignment Version 2 – I also gave detailed feedback as posted on the Discussion Board.
  • Assessment 3 – I wanted to give detailed feedback to Sandra W and Pam but it had already been given by two classmates. So I gave the feedback as attached Feedback to Sandra W n Pam’s Assessment 3. It is also posted on the Discussion Board.

References

E-Learning Guidelines. (2008). Retrieved 20 December, 2011, from http://elg.massey.ac.nz/Guidelines-questions.pdf

Reeves, T. C. & Hedberg, J. G. (2003). Interactive Learning Systems Evaluation. Englewood Cliffs: Educational Technology Publications.

Posted in Uncategorized | 1 Comment

Reflections on Assessment 3

Sunset at WeymouthAs this is my last blog, I’ve chosen the pic in my first blog (which is also one of my favourites) to insert above. It sets the scene well for at the end of a long day or process it is nice to put your feet up and enjoy the view or the outputs and outcomes you have created.

Mark, Tom and I met several times this week to work on Assignment 3 which we put together. It was a very fruitful session as together we brainstormed and collaborated on working on completing this Assessment. Our target was to complete it as soon as possible as we all wanted to relax and enjoy a well deserved holiday.

There was certainly a lot to do as we extracted what was required from the evaluation plan for the report. We worked together on the Executive Summary, Conclusions and Recommendations and slotted in our individual parts for the Discussion section. I had worked on Evaluation Q1, Mark on Evaluation Q2, and Tom on 3. I also did the overall summaries for the Teacher and Expert users and the Comparison between the two groups. Mark and Tom gave their valuable input and together we finalized it. There was a lot of camaraderie and as a team we were happy to have finally put it all together. When we reached the Conclusions … section, we were so tired as we had been working the whole day that we just ran out of steam and so I felt Bronwyn’ s suggestion to have ‘indepth summary of the evaluation outcomes’ was justified. Since then, we’ve been emailing each other updating on what needs to be done.

What was good about our team was that we each brought different areas of knowledge, skill and expertise to the project. This blend of strengths and weaknesses became self balancing with Tom lending his wisdom and experience and Mark contributing valuable feedback and his excellent skills in Excel, Word and formatting. We supplemented, supported and endorsed the overarching origination of the plan and like strands of wool together we wove a tapestry.

If someone were to ask me what is evaluation I would say in a nutshell it is about identifying what works and what doesn’t and why and how to improve it further. I was familiar with needs analysis, course and lecturer evaluation and summative evaluation.  After doing this course, I’m more confident about conducting a formal formative evaluation in future.

I do see the need for theories on which to anchor practice. For example the mixed methods approach to evaluation enables a combination of both qualitative and quantitative analysis so that the results are more valid and reliable. Also you get rich data which can be used to inform logical, rational and well informed decisions. Upon reflection personally and professionally, we need to evaluate if we are to do things better in terms of being more effective and efficient. To me including aspects of evaluation as part of the initial design and development process of a tool or online course was very insightful as evaluation contributes to its  ongoing development. I found the e-Learning Guidelines (2008) useful as they helped prompt the evaluation questions. I’ll certainly consider using them for future projects.

To me the whole process was engaging and stimulating. I have learnt lots from the literature, my interaction with my peers and Bronwyn and Bronwyn’s very detailed feedback on our assessments. What was different was working in a group to complete the assessments. That to me was a challenge as I do prefer to work on my own as I set my own goals and work towards achieving them. Working together with Tom and Mark has been a very good experience as we established a common goal to hand the work on time and to produce terrific work. It was a pleasure working together 🙂

I gave feedback to Sandra W and Pam’s draft evaluation report Feedback to Sandra W n Pam’s Assessment 3

Reference

E-Learning Guidelines. (2008). Retrieved from http://elg.massey.ac.nz/Guidelines-questions.pdf

Posted in Uncategorized | 1 Comment

Reflections on Assessment 2.1

You may have noted that I often have colourful pictures of sunsets, flowers and beaches in my postings and the reason is that I love nature and the bright, vivid colours tend to energise me as I write these blogs.   I do enjoy looking at them 🙂

When we submitted Assessment 2 (as blogged previously), I thought whew that’s over and done with but I was sadly mistaken as it came back and we had to ruthlessly cut it down. I think we had done a thorough analysis but we had a lot of pages and we were asked to cut this right back so now we’ve ruthlessly pruned the content. Mark, Tom and I did the pruning and Mark did the graphs which I must say looks impressive 🙂 In addition, the formatting had to be redone and that as you know is another time consuming process.

In a sense it was easy to do as we had all the details. If we had not been as thorough then it would have been more difficult to go back and redo the analysis. So in a sense I’m satisfied with the outcome. We do have a lot of data and I felt that I was not doing the analysis justice by reporting just the main findings as it is the details that flesh out the results.

As the analysis was detailed and thorough, I summarised the results at the end of each evaluation question to make it reader friendly. This though added to the length and the summary of the findings was presented at the end of the each sample group. I also thought that repeating the research questions and answering them with the findings was more reader friendly as there’s a lot of information to follow for the reader. But this admittedly also adds to the length of the results and discussion section. And may be seen to be repetitive as well.

I had organised the data in tables as this was easy for me to analyse.But I do take on board the suggestions to make the presentation of the data more attractive using charts, graphs and diagrams. So this was useful feedback for me as it is important to consider the reader when writing up the results. The clearer and more explicit you make your findings, the less questions your reader has for you. Also I need to update my skills working in excel as  churning out charts and figures using them is just fabulous.

In writing up the results, I had followed Debra Maddock’s exemplar as it too was on formative evaluation. I presented overviews to show how the findings would be reported, and tables relating the evaluation questions, sub-questions, questionnaire and interview responses. I had thought that this made the report more reader friendly as it indicated to the reader what to expect in the following sections. But this also increased the length and so had to be chopped.

What I’ve gained from this pruning process is to keep things short and simple. But this depends on the depth and breadth of the analysis. I wanted to  do justice to the data collection, analysis and reporting of results process and so I did exactly that. But then in the midst of all that I had forgotten about the reader. It is so important to use strategies to make your writing reader friendly.

Well we finally submitted Assessment 2 for marking.

Posted in Uncategorized | 1 Comment

Reflections on Assessment 2

Finally after what seems like a long haul, Assessment 2 has been submitted to Bronwyn. Sometimes you can’t see the wood for the trees and many thanks to Brownyn for pruning back the branches 🙂

Anyway it’s all done and here are my contributions and reflections on Assessment 2:

  1. Initially we divided the Evaluation Questions …Tom said he was interested in working on usability so he took EQ 2a and 2b, Mark did EQ1b and I did EQ1a. As I looked at the requirements of the assessment, I worked on the data analysis procedure and rationale. Also I extracted from the raw data what we needed to answer the evaluation questions and presented them in the appendices.
  2. After Bronwyn’s feedback, I starting incorporating the changes and revised how the data was presented. As I wrote the findings it was very interesting to see how the story unfolded in relation to the evaluation questions. It was very exciting for me personally as I now have the results of a formative evaluation on the iLP 🙂
  3. I did my part plus I also did Mark’s as his father was gravely ill for the last few weeks and passed away last week. It has been a very difficult time for him and Tom and I were happy to do what we could to help out. Tom revised his section and I then massaged it to make it consistent with all the sections. Tom very kindly offered to do the tables and that fit in well with the rest of the work. So we both worked through the week and weekend and with emails flying as Tom has said all the sections were completed. Mark did the tables, graphs and formatting and slotted in the appendices.
  4. Communication is absolutely crucial when working in a group. Our weekly meetings are great and even during holidays we made it a point to meet. Also participating in the Wimba sessions was extremely useful as we got immediate feedback on our work. Attending it with Tom made my task easier as he would ask questions and make comments which I had forgotten or had not thought of asking. So we got more feedback on our work.
  5. I read through Bronwyn’s feedback to ensure that all the comments and feedback had been incorporated. We also considered feedback from Sandra H, Fred and Rosanne in the assignment.
  6. What have I learnt from this exercise – One is that it’s important to conduct a pilot using a few samples. Yes I know that’s a luxury and in our case certainly not possible nor practical considering the time frame. The findings indicate that further probing of the responses from the samples might have elicited much richer data. Also knowing now what the responses are, slightly modified questions too might have elicited well possibly different findings.
  7. I gave feedback to Sandra H and Rosanne’s Assignment Version 2.
  8. Link to Assessment 2
Posted in Uncategorized | 3 Comments

Whew … submitting Assignment 2 for feedback …

Wow what a weekend that was… It was as usual busy … I have gotten into the habit of spending the whole of Saturday and most of Sunday working on the assignments. It surely helps that Dene is extremely understanding and supportive 🙂

So I started my work in the morning and got stuck into the analysis. As I trawled through the data, I became excited to find that the  findings on the evaluation of the iLessonPlan seemed to be rather positive 🙂 Anyway more on that later at the end of the evaluation …

The first task was to make sense of the raw data we had collected using a questionnaire in Google forms and an interview. As I said in my previous blog, the data in Google forms was copied into an excel spreadsheet and that certainly helped me to identify the trends much better.

Ordering the responses using established criteria is also essential to making sense of the data. I played around with several versions always keeping in mind the evaluation questions. This  of course was time consuming but the more  I looked at it the better I was able to identify the trends and patterns.

Then I flicked emails to Tom and Mark updating them of the progress. Tom and Mark were working on their sections and we kept in touch constantly about what was happening as we set the deadline for Monday to submit the assignment for feedback. As Mark said once something to the effect of, ‘Alages forward our emails to Bronwyn and she’ll have a better idea of what we’re doing as we only post our final draft/s on the wiki’.

Gradually my section is taking shape and I can see the end … well almost … Monday arrives and the internet is unavailable from 2 pm onwards … oh dear 😦

Tom does a whole lot of work and it’s lost in cyberspace 😦 With blood pressure rising, he manages to save it … youuuuuuuuuuuhoooooo. I do some work and the file is lost again in cyberspace … hmmm surely it’s NOT Friday the 13th??

Anyway all’s well as when we met this afternoon we put our sections together … gosh what a relief! It’s still a work in progress but it is ready for feedback now 🙂 It’s a day later than when we said we’d submit it but better late than never 🙂

It’s great working with Tom and Mark. We make a good team 🙂

Posted in Uncategorized | Leave a comment

Our meeting last Thurs

I wrote this last Fri (11/11), saved it as a draft and then forgot to ‘publish’ as I got busy with my section of Assessment 2. So here it is …

Finally all the data has been collected. Despite our plan to complete this in 2 weeks, it has taken us about 3 1/2 weeks. So this has delayed our data analysis which we plan to complete by the end of this week.

Also working with Google forms has proved to be quite a challenge. While it is relatively simple and straightforward to create questionnaires using Google forms, it is important to ensure that you know how to use the different functions in it well enough to extract the data that you need. I had planned to learn more about it last weekend but due to the slow internet service, downloading tutorials from youtube proved to be such a long and painful process. No matter the issue was resolved by copying the data into excel, flipping it and then filtering the data into the two sample groups: Teacher and Expert users.

At today’s  weekly meeting, Tom, Mark and I revisited the evaluation questions, sub-questions and the data collection instruments: questionnaire and interviews using a table I had done.  By relating the responses to the questionnaires and interviews to the evaluation questions, I felt it was easy to answer the questions.

More later 🙂

Posted in Uncategorized | Leave a comment

Life happens

On Thursday at our weekly meeting, we sorted out who’s doing what in relation to writing up of the findings. It was best to do it in sections relating to the evaluation questions. Tom indicated his interest in doing the section on usability and Mark and I then took the other sections. Tom’s section is relatively bigger and so we said we’d pitch in to help him if need be.

Generally collecting the data was a smooth and efficient process as Tom had identified how we were to proceed and Mark and I added our input. We also agreed to write up our notes on how the trailling of the iLP went as it was still fresh in our minds.

Most of my samples were available during the times we had agreed on. But then despite me making arrangements 2 weeks earlier, one sample has proven to be a bit elusive and so I’ll be doing mine on Monday. So this is where plans are important as contingencies have to be in place for these types of situations. So our analysis for now has been delayed:-(

So note to myself … in future ensure that a bigger sample size is identified and if there is an attrition rate, then I’ll still have some decent numbers to work with.

Then this morning we began to have extremely slow internet service. Dene has complained to Telecom and we have been assured that it will be restored in 2 days’ time. Oh dear … as I do most of my work on Saturdays and Sundays, this is greatly hampering my progress. Technology is great when it works and well not so great when the downloads take you ages … Oh well back to the drawing board …

So Plan 2 is to start working on incorporating Bronwyn’s feedback on my sections of the evaluation plan to be included into the report later. Moreover, I’ll be looking at a template we can use to present our results and findings. Good thing I had downloaded some articles earlier, so I hope to read those and make the relevant notes for the section on discussion.

Personally I do my best to meet deadlines but then life happens and so options are good. Also what I take away from this experience is not to lose my sanity when it matters most… keep my wits about me and continue to do the best I can.

Posted in Uncategorized | 2 Comments