From: IJAIED Editor [ijaied@paulbrna.demon.co.uk] Sent: 04 February 2005 00:21 To: b.galitsky@dcs.bbk.ac.uk Cc: IJAIED Editor Subject: IJAIED406: Results of your submission International Journal of Artificial Intelligence in Education --------------------------------------------------------------------- IJAIED406: A simulation tool that improves autistic reasoning about mental attitudes Dear Boris Galitsky, Reviews of this paper which you submitted to IJAIED are included below. As you can see from the comments, the paper is seen as needing significant improvements before a decision to publish in IJAIED can be given. However, it seems to me that the reviewers (and myself) are open to being persuaded that this is a valuable contribution but it is not possible to determine how valuable in its present form. If you feel that you would like to resubmit, please revise your paper, paying careful attention to the comments provided by the reviewers. When you send in your revised paper, it would be appreciated if you could say in a covering letter how you have responded to the reviewers' comments. This should speed up a final decision either way. To format your paper, you might find it useful to follow the guidelines given on the IJAIED WWW pages: http://www.ijaied.org I look forward to receiving your revision. Meanwhile, if I can help further please let me know. All the best Paul Editor, IJAIED ----------Review 1------------------------------------------------------------ IJAIED Reviewer Form -------------------- Paper No: IJAIED406 A simulation tool that improves autistic reasoning about mental attitudes Comments (which will be sent directly to the author) -------- 1. Is the subject of the paper suitable for IJAIED? It could be - at the moment, it lacks sufficient clarity. 2. Is the content of the paper likely to be of interest to and appropriate for IJAIED readers? Potentially, yes. 3. Is the paper technically sound and accurate in its AI and Education content? Difficult to tell - key description seem to be missin/difficult to follow 4. Is this a new and original contribution? Does the author make clear what this contribution is? I think there may well be a very interesting contribution to the study of autism but it is obscured by the style of the paper - also, what kind of understanding does it bring to the study of systems that help autistic children - either in terms of representation or of interaction? 5. Are the major claims and conclusions substantiated? Have the ideas or systems been tested or evaluated sensibly? Broadly yes - but there have to be questions about the methodology. 6. Is the paper clear, explicit, and well-organised? Is the length appropriate for the content? Are there any gaps or redundancies? Sewe below 7. Are the title and abstract informative? OK 8. Does the paper adequately refer to related work? Are the references complete and necessary? Not really - there are references to work other than the authors but these seem to have limited impact on the work. 9. Overall, is the paper acceptable? Please rate on a scale 1 - 9, as follows: 9 - accept as it is 7 - accept with minor revisions 5 - needs major revision before it could be accepted 3 - needs major revision but may not be acceptable even then 1 - not acceptable 3 10. Any other general comments or specific suggestions for the author? This is a very promising contribution but there are a number of problems with this - some of a very general kind, and some relating to the needs of the Journal and its audience. First, the general issues 1. Many terms are not explained well enough for an audience not as versed in theories of the nature of autism, the developmental processes that might compensate for genetic (or other) damage, and theories of mind. For example a. references to previous work needs to be "unpacked" eg ways in which autistics reason about mental states and making it clear any distinctions between autistics and those with "mental disorders" in general. b. There is a significant difference between theorising about the reasoning of an autistic person and theorising about the developmental path which might be taken by the autistic person. The paper suggests that there are reasonably detailed theories in existence to allow autistic reasoning to be modelled to some extent - but where are the models that explain how an autistic can improve? There appear to be some suggestions based on personal experience - but is that all is available? Since I think experience is important it would help to know if other significant research has been done which comes to broadly the same result - i.e. the autistic person finds it easier to learn a kind of formal reasoning approach to social contact, reasoning about other minds, understanding and expressing emotion etc. c. The methodology of the approach is a little unusual - firstly, in mainly drawing upon the work of the author relating to autism and secondly, in approach taken to the empirical work (both quantitative and qualitative). d. There is a need for this paper to be revised by a native english speaker - many grammatical constructions are not clear or wrong. Second, the issues connected with the Jorunal and its readership: 2.1 It is not at all clear what kind of paper this is. It seems at times as if this paper is about an existing delivery system targeted at providing treatment (of a kind) for autistics. At other times it seems to be about the underlying technologies needed to achieve such an end. If it is both then it might be more useful to distinguish the underlying technologies from the delivery platform. To achieve this, it would be necessary to devote space to the exposition of the computational context, explaining in more detail what kind of environment NL_MAMS is. 2.2 Please make it clearer in the paper that NL_MAMS is the name of engine that does the inference rather than the name of the delivery system for the autistics. (Does the delivery system have a name?) 2.3 When describing the empirical work, even though the paper is not setting out to explore some hypothesis testing scenario, it would help to provide a more standard frammework. For example, who was selected for working with the system, how they were selected, how they were instructed to work with the system, what the autistics "did" with the system, how the data was collected and how it was interpreted. This might make the paper longer but significantly clearer. 2.4 If the paper is within the scope of IJAIED then it would help to understand the implications of the approach - but this is very difficult to see at present. ----------Review 2------------------------------------------------------------ IJAIED Reviewer Form -------------------- Paper No: IJAIED406: A simulation tool that improves autistic reasoning about mental attitudes Comments (which will be sent directly to the author) -------- 1. Is the subject of the paper suitable for IJAIED? Yes, I think so...potentially... 2. Is the content of the paper likely to be of interest to and appropriate for IJAIED readers? Yes - again, potentially... 3. Is the paper technically sound and accurate in its AI and Education content? The paper describes a 'natural language multiagent mental simulator' (NL_MAMS) which, the author suggests, is useful for teaching autistic children to reason about the mental states (emotions, beliefs, attitudes) of other people. The author's hypothesis is that autistic children are likely to benefit from a learning environment in which `axioms' for discerning the mental states of others can be acquired quite formally (ie. deductively, in terms of formally specified assertions and the semantic relations between them). In this respect autistic children differ from non-autistic children - the latter acquire socio-emotional knowledge via more `everyday' modes of learning such as induction over cases, by being told, etc. 4. Is this a new and original contribution? Yes - it is interesting and original in that the types of inference and reasoning that AIED systems such as NL_MAMS are strong in are, in this application, so well matched to modes of reasoning that are effective for teaching autistic learners. It is also interesting that the domain is that of socio-emotional skill acquisition - not often addressed in AIED studies... Does the author make clear what this contribution is? Here I think the paper needs substantial revision in order to clarify the nature of the contribution. The paper describes autistic reasoning, and outlines eight 'steps' (stages? components?) of mental 'actions' and how these can be specified as 'mental formulae' of knowledge-belief-intention units. These units can be represented in natural language and logic. As it stands the sections setting out the 'steps' tends to dominate the paper - maybe change the title and abstract to reflect the importance of this section? The NL_MAMS system is also described. It would help the reader if actual screen shots of the system could be used instead of the hybrid screen shot/table in Fig 2. It is not clear what the autistic student user *actually* sees initially, how s/he interacts with the system and how the system's output is presented and interpreted by the user. Perhaps an actual step-by-step, detailed, walkthrough of a sample interaction by an autistic child with NL_MAMS could be presented. 5. Are the major claims and conclusions substantiated? Have the ideas or systems been tested or evaluated sensibly? The evaluation of NL_MAM (the case studies) do not convincingly demonstrate its effectiveness. Case studies of 2 students (Alexandra and Leon) are presented and changes from 2001 to 2002 are indicated (Table 1). However it is unclear to what extent these changes can be attributed to those students' interactions with NL_MAM since the study is uncontrolled and the students received a host of other therapeutic interventions also. The criteria and protocols by which the behavioural changes were assessed is also unclear. 'Key success features' for 7 subjects are presented in Table 2 but details about how these behaviours were assessed are not provided. Were they the subjective judgements of the author, of special needs teachers ? These points also apply to the 'before' and 'after' data - much more detail regarding the evaluation's methodology, behavioural assessment techniques, details of NL_MAMsintervention (duration, sessions, etc), control condition, etc are required. Statistical analyses of the control student/experimental student differences should also be conducted. 6. Is the paper clear, explicit, and well-organised? Is the length appropriate for the content? Are there any gaps or redundancies? I found the paper very difficult to read. The author should have the revised paper proof read for grammatical and typographical errors, as well as English expression. There are very many examples of poor English expression and grammar - eg "...is a good assistance to parents' and the paper is riddled with spelling errors - too many to enumerate... 7. Are the title and abstract informative? Title makes 'improvement' claim which is not substantiated by data presented in paper - title could be changed to reflect emphasis on 'steps', description of NL_MAMS and its rationale (see comments above). Abstract rather brief - seems incomplete - it ends '..and analyze the results of' ??? 8. Does the paper adequately refer to related work? Are the references complete and necessary? To the best of my knowledge. 9. Overall, is the paper acceptable? Please rate on a scale 1 - 9, as follows: 9 - accept as it is 7 - accept with minor revisions 5 - needs major revision before it could be accepted 3 - needs major revision but may not be acceptable even then 1 - not acceptable I would assess this paper as 4 in its present form. ----------Review 3------------------------------------------------------------ IJAIED Reviewer Form -------------------- Paper No: 406 Comments (which will be sent directly to the author) -------- 1. Is the subject of the paper suitable for IJAIED? Yes. The paper describes the "mental state simulator", an AI program which uses formal reasoning to teach children diagnosed with high functioning autism about mental state concepts and representations. The application is novel and the results with a limited groups of children are promising. 2. Is the content of the paper likely to be of interest to and appropriate for IJAIED readers? definitely yes. The paper is an original application of AI formal reasoning to the exciting area of autism education/therapy. 3. Is the paper technically sound and accurate in its AI and Education content? to some extent. The authors present a novel approach to teaching children with HFA about mental states at different levels (i.e. simple and complex). However, the paper only addresses a limited range of mental states, and it is hard to immediately see how it would generalize to other types of mental states (e.g. cognitive states / affective states). Also, it seems quite theoretic and the children may have difficulties in the ability to generalise from it to everyday life (unless the expectation is for children to run these decision trees in their heads). The paper would benefit from having such a discussion. 4. Is this a new and original contribution? Does the author make clear what this contribution is? The mental state simulator is a novel method for teaching children with autism how to understand mental concepts. The description of the contribution has to be distilled from the paper, and needs to spelt out in a simple and direct way. 5. Are the major claims and conclusions substantiated? Yes Have the ideas or systems been tested or evaluated sensibly? to some extent. details: The system has only been tested on two children, so it is hard to draw conclusions on the usability and generalizability of the system beyond these two cases. However, it is important to note that it in the field of autism it is hard to find many cases and conduct trials that extend over such a long period of time. 6. Is the paper clear, explicit, and well-organised? Is the length appropriate for the content? Are there any gaps or redundancies? Perhaps this is the biggest problem with this paper. The paper was very hard read and to follow. Perhaps a simpler description of the technical details could be added with each section to describe the big picture / idea behind that section. 7. Are the title and abstract informative? yes 8. Does the paper adequately refer to related work? Are the references complete and necessary? More references should be added re: the theory of mind theory of autism. 9. Overall, is the paper acceptable? Please rate on a scale 1 - 9, as follows: 5 9 - accept as it is 7 - accept with minor revisions 5 - needs major revision before it could be accepted 3 - needs major revision but may not be acceptable even then 1 - not acceptable 10. Any other general comments or specific suggestions for the author? Since the content of the paper is inherently inter-disciplinary, namely drawing upon formal reasoning in AI, theory of autism, education, I think the paper would benefit a lot from being re-written so that it is accessible to people who are not by necessity familiar with all three research domains. For instance, with respect to the AI community, more background should be given about theory of autism (the autism spectrum of conditions), mental state concepts, and the different classes within those, and other (education) methods for autism therapy. The author mentions some "peculiarities" in reasoning of people with autism that are not described further, so as it stands its hard to conceptualize exactly how is the use of mental state concepts in autistic people different from normal ones. To make the paper somewhat accessible to pyschologists, or professionals/pracitioners who work with children with HFA (which are really the potential users of the mental state simulator), the technical details of the paper would benefit from being re-written so that it is accessible who are perhaps not familiar with formal reasoning. For example, its not clear what the traditional modal logic-based formal representation is (page 2). The correct citation for the mind reading software is : Baron-Cohen, S., Golan, O., Wheelwright, S., & Hill, J. J. (2004). Mind Reading: the interactive guide to emotions. London: Jessica Kingsley Limited (www.kjp.com ). ----------------------------------------------------------------------