Haowei's Diary

Diary

plan to use an iterative approach at the last week towards finalising the report in order to shorten the feedback loop from Stuart. send a draft for feedback iteratively on 13th, 15th, 17th and 19th of June

(7 Jun - 13 Jun)

• got hold of the user study DVDs
• started analysing the user study data (videos, questionnaires, user tasks)
• met Stuart got a lot of advice on the final report

(31 May - 6 Jun)

• competed the 2nd day of user study with another 4 users
• continued drafting the final report

(24 May - 30 May)

• did a pilot study in OptiPortal with Stuart, thanks to Roger who's very helpful
• completed the 1st day of user study with 4 users
• started drafting the final report

(17 May - 23 May)

• met supervisors
• visited the OptiPortal usability lab to have a look at the settings and available equipments

(10 May - 16 May)

• met supervisors, discussed the user study design
• created tasks form, time-slot form

(3 May - 9 May)

• revising softvis paper, final to be submitted by 11pm 9 May

(26 Apr 2010 - 2 May 2010)

• sent out 3 drafts of the softvis paper, had feedback from Craig
• attending 2 user studies this Friday

(19 Apr 2010 - 25 Apr 2010)

• working on the SoftVis'10 paper, intend to have draft 1 ready by Friday.

week7 (12 Apr 2010 - 18 Apr 2010)

• determined to write up a paper for SoftVis'10 without the user experiment component
• James gave feedbacks on some JHotDraw viz
• started working on the SoftVis'10 paper

week6 (5 Apr 2010 - 11 Apr 2010)

• meeting Stuart - revising the HEC application
• gained some more insights through JHotDraw viz

week5 (T1 29 Mar 2010 - 4 April 2010)

• continue to draft the paper for SoftVis conference
• Met James and Stuart on Tuesday, discussed further in terms of the experiment setup, we felt time is tight for carrying out the experiment if we aim for the end of April paper submission.
• writing up the information sheet for HEC
• writing up the consent form for HEC
• used JavaParser to extract Package, Import, Class, Interface, Public, Private, and Protected methods
• exploring JHotDraw with the above Java entities identified

week4 (T1, 22 Mar 2010 - 28 Mar 2010)

• Met James briefly discussing ideas of the user study, mentioned selection of subject dataset would depend on domain expertise because the insight-based evaluation approach would require rating of discovered insights by domain expert; James pointed out JHotDraw is a potential dataset as he is familiar with it and there are plenty of papers talking about JHotDraw
• filled in the HEC application form and gave to Stuart
• attended Craig's PhD proposal dry-run, very useful for now and future.
• Started drafting the paper for SoftVis conference
• Started designing a controlled experiment with open-ended protocol, which helps to discover, and measure based on quantifying insights.

week3 (T1,15 Mar 2010 - 21 Mar 2010)

• sought guidance from John and James
• meeting with Stuart
  • talked about how i want to move from evaluating how effective Jigsaw aids "program maintenance" (benchmark task) to "program discovery" (insight-based) evaluation, according to a paper read, "program maintenance" often uses maintenance tasks based evaluation - easy to measure tool effectiveness based on completion of tasks and precision, "program discovery" often uses insight-based evaluation - hard to measure tool effectiveness as you don't know what you are looking for. After reading the paper "An Insight-based Methodology for Evaluating Bioinformatics Visualizations", i felt it's probably a more appropriate approach to evaluate Jigsaw visualizing software.
  • discussed some ideas of user experiment design (e.g. evaluate Jigsaw with users of different experience levels of different software systems to allow variables to be observed between different setups, 3 programming language experts Jigsawing JRuby, 3 2nd year students Jigsawing JRuby, professional Ruby developers jigsawing JRuby, etc.)
• literature survey focused on insight-based evaluation
• read "Usability Evaluation Considered Harmful (Some of The Time)"
• explored past BELIV, SoftVis, VISSOFT, ICPC papers, read ones of relevance, compiled TOC in reference page

week2 (T1, 8 Mar 2010 - 14 Mar 2010)

• literature review on SoftVis evaluations based on software comprehension tasks - measuring how well typical software comprehension questions are answered through aid of visualizations

week1 (T1, 1 Mar 2010 - 7 Mar 2010)

• read a dozen of papers on SoftVis, InfoVis, Visual Analytics evaluations, still not to sure how to evaluate using visual analytics tool to conduct software visualization; for example, how to decide what heuristics to use, do i use VA heuristics, InfoVis heuristics, SoftVis heuristics, or combination of them, another problem is, there don't seem to be a well recognized/established set of heuristics in this area, then how do i rationalize on using these ones not the others, what criteria do i use to select heuristics, etc and etc .
• met Stuart and James with the above questions, got some guidance generalized as below:
  • for heuristics approach, dig out the most influential papers on SoftVis evaluation and use those ones
  • for user study approach, find the common software comprehension questions and see how well they are answered using the VA tool, in comparison with the existing study (documented in influential papers, or papers by influential figure)
  • usability heuristics, maybe???
• formulated a strategy after a weekend of study
  • evaluate VA tools using SoftVis criteria, focus on software comprehension tasks, heuristics, user studies, expert reviews
  • then evaluate VA tools using extra criteria such as knowledge discovery (best paper - "A Knowledge Task-Based Framework for Design and Evaluation of Information Visualizations"

week ?? (22 Feb 2010 - 28 Feb 2010)

• performed more visual analysis on JRuby using Jigsaw with "Class", "Package", "Method" and "Email" entity
• studied papers on SoftVis evaluation, evaluation of visual analytics environments
• Demonstrated Jigsaw visualizing JRuby-1.0.1 source code at the ELVIS meeting

week 12 (15 Feb 2010 - 21 Feb 2010)

• revised and submitted milestone1 report
• introduced "Package" entity to Jigsaw visualisation
• wrote a Java program using JavaParser to extract public methods from Java source code
• introduced "Method" entity to Jigsaw visualisation
• studied "A Task Oriented View of Software Visualization" and "Classifying Desirable Features of Software Visualisaztion Tools for Corrective Maintenance"
• end of T3

week 11 (8 Feb 2010 - 14 Feb 2010)

• Quite happy with what Jigsaw can do visualising software, and John Stasko is interested to be more involved with our research.
• met Craig, James, Stuart to get feedbacks on the milestone1 report, decided to submit paper to SoftVis instead of VL/HCC.
• St Valentines's Day and Chinese New Year are on the same day ... that's convenient!

week 10 (1 Feb 2010 - 7 Feb 2010)

• use customized list to identify entity "Class" for Jigsaw, i am getting some interesting aspects of JRuby-1.0.1, Ant-1.7.1 and Tomcat-5.5.17
• getting big help from John Stasko in regards to the use of Jigsaw, i think we are getting somewhere with Jigsaw
• consolidating experiments performed with Jigsaw and Spotfire into words, didn't realise until now that writing about it actually made me learn more about the tools
• milestone 1 report is taking shape

week 9 (25 Jan 2010 - 31 Jan 2010)

• started writing the "introduction" chapter of milestone 1 report
• after realising the given metrics CSV contain errors, started doing my own metrics CSV extract using Scitools Understand 2.0
• understand more about Understand 2.0 with the metrics it produces, decided to do some verifications on the metrics produced rather than blindly trusting the tool
• used Scitool Understand to generate OO metrics on JRuby, Ant, Argouml, Hibernate Core, Tomcat, perform visual analysis on them using Spotfire treemap, scatter plot, pie chart, line chart, etc.
• used Scitool Understand to generate architecture dependency metrics on Ant-1.7.1, perform visual analysis on it using Jigsaw

week 8 (18 Jan 2010 - 24 Jan 2010)

• Using Spotfire to perform visual analysis of Ant-1.1 OO metrics - not exactly knowing what hypothesis can i make when interacting with the visuals
• Thus, getting into some OO metrics literatures to gain more knowledge about the metrcis
• reading "Object Oriented Metrics in Practice"

week 7 (11 Jan 2010 - 17 Jan 2010)

• start using TexShop, BibDesk with CiteULike, it's really cool ^_^
• met James and Stuart after ELVIS - get hands dirty with the tools is basically the msg i got
• Used Jigsaw to analyse Ant-1.1 source code (unstructured data), the entities identified do not seem making much sense; intended to use Jigsaw to analyse structured data but the XLS import functionality isn't working - John Stasko said will release a new version next week which should fix the issue.
• Used Spotfire to visualise OO metrics produced SciTools Understand; getting into OO metrics literatures to gain more understanding on meaning of metrics and their application - this should help me interacting with Spotfire visuals in a more sensible manner.

week ? (4 Jan 2010 - 10 Jan 2010)

• started milestone1 report draft
• started to focus on establishing a set of heuristics for evaluating visual analytics tool
• started learning LaTeX
• downloaded and installed Spotfire

week ? (28 Dec 2009 - 3 Jan 2010)

holidays

week6 (21 Dec 2009 - 27 Dec 2009)

• literature survey and playing with Jigsaw

week 5 (14 Dec 2009 - 20 Dec 2009)

• installed Jigsaw, experiencing "Entity Identification" problem with two bundled-in modules, applied a key for using Calais web service for entity identification
• installed Tableau (14 days trial), very easy to use with the example given, and plenty of video tutorials, tried to use our own data experiencing data source connecting problem
• literature surveying - found Moose platform and CodeCity on top of it, it is FASINATING !!!
• installed the corpus on my macbook, installed Jigsaw and scitools' Understand 2.0
• studying in CO251 on Friday; had my first ELVIS meeting
• read Diehl's "Software Visualization"

week 4

12 - 13 Dec 2009

• Looked at John Staskos VLHCC 2008 keynote: http://www.cc.gatech.edu/gvu/ii/talks/vlhcc-softvis08.pdf
• Looked at John Staskos Softvis 1999 workshop keynote: http://www.cc.gatech.edu/gvu/ii/talks/softvis99.pdf
• read the book "Software Visualization - Visualizing the Structure, Behaviour, and Evolution of Software"

11 Dec 2009

• meeting with Craig, got Corpus DVD, got a lot of guidance

10 Dec 2009

• exploring Prefuse
• read "Visualising the Word Structure of Java Class Names", "Towards End-User Web Software Visualisation"

9 Dec 2009

• still mucking with Improvise, but not getting anywhere ...
• started trying Prefuse see if it's me or the tool

8 Dec 2009

• finalised project proposal

week 3

6 Dec 2009

• have been playing with Improvise the whole weekend (well almost), it's not a particularly easy or intuitive tool to work with
• read various other papers in search of what aspects of software are being measured or visualised ...

2 Dec 2009

• working on project proposal first draft
• read "A Task Oriented View of Software Visualization"
  • key argument: no single software visualisation tool can address all software engineering tasks simultaneously.
  • tasks: coding, debugging, design, re-engineering, software evolution
  • should i set a task for visualisation? say visualising software changes

1 Dec 2009

reading about "Improvise"

• a conceptual model of multiple view coordination - Live Properties
• a visual abstraction language - Coordinated Queries
• Metavisualization

30 Nov 2009

read Craig's "PhD Research Proposal - Visualising Software Corpus Analysis"

read "Understanding the Shape of Java Software" - the motivation section resonates with the project

29 Nov 2009

started reading "Illuminating the path"

week 2

26 Nov 2009

first meeting with James and Craig

discussed research topic, direction, readings

read Craig's "Towards Visual Software Analytics"

Conference key dates

SoftVis

• Papers Due : April 30, 2010
• Notification : June 30, 2010
• Posters/Tool Demos Due: July 14, 2010
• Symposium : October 25-26, 2010

VL/HCC

• Workshop/Tutorials:
  • informal "intention to submit" email: 4 January 2010
  • submission of proposals by email: 11 January 2010
  • notification of acceptance: 1 February 2010
• Abstract submission: 22 February 2010
• Paper submission: 8 March 2010
• Notification of decision: 18 May 2010
• Camera-ready copy: 14 June 2010

Some guiding questions

• What parts of the "software" are you planning to focus on? (OO metrics, source code)
• Are you going to look for some specific programming language construct or feature, e.g. Big Java classes, use of inheritance, use of fields, overriding, dependency cycles etc.?
• How do you plan to get the software data into these systems?
• Are you going to parse the source code or byte code from the corpus?
• How does one evaluate visual analytics tools?
• What criteria are you going to use to evaluate the VA tools?
• What features of the tools are you going to evaluate?
• Do you plan to do some user testing of these tools or will these be expert/heuristic (i.e. you) evaluations?
• What methods will you use to measure the users using the VA tools?

Literature Survey

• Literature survey and references