From: IJAIED Editor [ijaied@paulbrna.demon.co.uk]
Sent: 04 February 2005 00:21
To: b.galitsky@dcs.bbk.ac.uk
Cc: IJAIED Editor
Subject: IJAIED406: Results of your submission
International Journal of Artificial Intelligence in Education
---------------------------------------------------------------------
IJAIED406: A simulation tool that improves autistic reasoning about
mental attitudes
Dear Boris Galitsky,
Reviews of this paper which you submitted to
IJAIED are included below. As you can see from
the comments, the paper is seen as needing significant
improvements before a decision to publish in IJAIED
can be given.
However, it seems to me that the reviewers (and myself)
are open to being persuaded that this is a valuable contribution
but it is not possible to determine how valuable in its present form.
If you feel that you would like to resubmit, please revise
your paper, paying careful attention to the comments
provided by the reviewers.
When you send in your revised paper, it would be
appreciated if you could say in a covering letter
how you have responded to the reviewers' comments.
This should speed up a final decision either way.
To format your paper, you might find it useful to follow the
guidelines given on the IJAIED WWW pages:
http://www.ijaied.org
I look forward to receiving your revision.
Meanwhile, if I can help further please let me know.
All the best
Paul
Editor, IJAIED
----------Review 1------------------------------------------------------------
IJAIED Reviewer Form
--------------------
Paper No: IJAIED406
A simulation tool that improves autistic reasoning about mental attitudes
Comments (which will be sent directly to the author)
--------
1. Is the subject of the paper suitable for IJAIED?
It could be - at the moment, it lacks sufficient clarity.
2. Is the content of the paper likely to be of interest to and
appropriate for IJAIED readers?
Potentially, yes.
3. Is the paper technically sound and accurate in its AI and Education
content?
Difficult to tell - key description seem to be missin/difficult to follow
4. Is this a new and original contribution? Does the author make clear
what this contribution is?
I think there may well be a very interesting contribution to the study of
autism but it is obscured by the style of the paper - also, what kind of
understanding does it bring to the study of systems that help autistic
children - either in terms of representation or of interaction?
5. Are the major claims and conclusions substantiated?
Have the ideas or systems been tested or evaluated sensibly?
Broadly yes - but there have to be questions about the methodology.
6. Is the paper clear, explicit, and well-organised? Is the length
appropriate for the content? Are there any gaps or redundancies?
Sewe below
7. Are the title and abstract informative?
OK
8. Does the paper adequately refer to related work? Are the references
complete and necessary?
Not really - there are references to work other than the authors but these
seem to have limited impact on the work.
9. Overall, is the paper acceptable? Please rate on a scale 1 - 9, as follows:
9 - accept as it is
7 - accept with minor revisions
5 - needs major revision before it could be accepted
3 - needs major revision but may not be acceptable even then
1 - not acceptable
3
10. Any other general comments or specific suggestions for the author?
This is a very promising contribution but there are a number of problems
with this - some of a very general kind, and some relating to the needs of
the Journal and its audience.
First, the general issues
1. Many terms are not explained well enough for an audience not as versed
in theories of the nature of autism, the developmental processes that might
compensate for genetic (or other) damage, and theories of mind. For example
a. references to previous work needs to be "unpacked" eg ways in which
autistics reason about mental states and making it clear any distinctions
between autistics and those with "mental disorders" in general.
b. There is a significant difference between theorising about the reasoning of
an autistic person and theorising about the developmental path which might
be taken by the autistic person. The paper suggests that there are
reasonably detailed theories in existence to allow autistic reasoning to be
modelled to some extent - but where are the models that explain how an
autistic can improve? There appear to be some suggestions based on personal
experience - but is that all is available? Since I think experience is
important it would help to know if other significant research has been done
which comes to broadly the same result - i.e. the autistic person finds it
easier to learn a kind of formal reasoning approach to social contact,
reasoning about other minds, understanding and expressing emotion etc.
c. The methodology of the approach is a little unusual - firstly, in mainly
drawing upon the work of the author relating to autism and secondly, in
approach taken to the empirical work (both quantitative and qualitative).
d. There is a need for this paper to be revised by a native english
speaker - many grammatical constructions are not clear or wrong.
Second, the issues connected with the Jorunal and its readership:
2.1 It is not at all clear what kind of paper this is. It seems at times as
if this paper is about an existing delivery system targeted at providing
treatment (of a kind) for autistics. At other times it seems to be about the
underlying technologies needed to achieve such an end.
If it is both then it might be more useful to distinguish the underlying
technologies from the delivery platform. To achieve this, it would be
necessary to devote space to the exposition of the computational
context, explaining in more detail what kind of environment NL_MAMS is.
2.2 Please make it clearer in the paper that NL_MAMS is the name of engine
that does the inference rather than the name of the delivery system for the
autistics. (Does the delivery system have a name?)
2.3 When describing the empirical work, even though the paper is not
setting out to explore some hypothesis testing scenario, it would help to
provide a more standard frammework. For example, who was selected for
working with the system, how they were selected, how they were instructed
to work with the system, what the autistics "did" with the system, how the
data was collected and how it was interpreted. This might make the paper
longer but significantly clearer.
2.4 If the paper is within the scope of IJAIED then it would help to
understand the implications of the approach - but this is very difficult to
see at present.
----------Review 2------------------------------------------------------------
IJAIED Reviewer Form
--------------------
Paper No: IJAIED406: A simulation tool that improves autistic
reasoning about mental attitudes
Comments (which will be sent directly to the author)
--------
1. Is the subject of the paper suitable for IJAIED?
Yes, I think so...potentially...
2. Is the content of the paper likely to be of interest to and
appropriate for IJAIED readers?
Yes - again, potentially...
3. Is the paper technically sound and accurate in its AI and Education
content?
The paper describes a 'natural language multiagent mental simulator'
(NL_MAMS) which, the author suggests, is useful for teaching autistic
children to reason about the mental states (emotions, beliefs,
attitudes) of other people. The author's hypothesis is that autistic
children are likely to benefit from a learning environment in which
`axioms' for discerning the mental states of others can be acquired
quite formally (ie. deductively, in terms of formally specified
assertions and the semantic relations between them). In this respect
autistic children differ from non-autistic children - the latter
acquire socio-emotional knowledge via more `everyday' modes of
learning such as induction over cases, by being told, etc.
4. Is this a new and original contribution?
Yes - it is interesting and original in that the types of inference
and reasoning that AIED systems such as NL_MAMS are strong in are, in
this application, so well matched to modes of reasoning that are
effective for teaching autistic learners. It is also interesting that
the domain is that of socio-emotional skill acquisition - not often
addressed in AIED studies...
Does the author make clear what this contribution is?
Here I think the paper needs substantial revision in order to clarify
the nature of the contribution. The paper describes autistic
reasoning, and outlines eight 'steps' (stages? components?) of mental
'actions' and how these can be specified as 'mental formulae' of
knowledge-belief-intention units. These units can be represented in
natural language and logic.
As it stands the sections setting out the 'steps' tends to dominate
the paper - maybe change the title and abstract to reflect the
importance of this section?
The NL_MAMS system is also described. It would help the reader if
actual screen shots of the system could be used instead of the hybrid
screen shot/table in Fig 2. It is not clear what the autistic student
user *actually* sees initially, how s/he interacts with the system and
how the system's output is presented and interpreted by the
user. Perhaps an actual step-by-step, detailed, walkthrough of a
sample interaction by an autistic child with NL_MAMS could be
presented.
5. Are the major claims and conclusions substantiated?
Have the ideas or systems been tested or evaluated sensibly?
The evaluation of NL_MAM (the case studies) do not convincingly
demonstrate its effectiveness. Case studies of 2 students (Alexandra
and Leon) are presented and changes from 2001 to 2002 are indicated
(Table 1). However it is unclear to what extent these changes can be
attributed to those students' interactions with NL_MAM since the study
is uncontrolled and the students received a host of other therapeutic
interventions also.
The criteria and protocols by which the behavioural changes were
assessed is also unclear. 'Key success features' for 7 subjects are
presented in Table 2 but details about how these behaviours were
assessed are not provided. Were they the subjective judgements of the
author, of special needs teachers ? These points also apply to the
'before' and 'after' data - much more detail regarding the
evaluation's methodology, behavioural assessment techniques, details
of NL_MAMsintervention (duration, sessions, etc), control condition,
etc are required. Statistical analyses of the control
student/experimental student differences should also be conducted.
6. Is the paper clear, explicit, and well-organised? Is the length
appropriate for the content? Are there any gaps or redundancies?
I found the paper very difficult to read. The author should have the
revised paper proof read for grammatical and typographical errors, as
well as English expression. There are very many examples of poor
English expression and grammar - eg "...is a good assistance to
parents' and the paper is riddled with spelling errors - too many to
enumerate...
7. Are the title and abstract informative?
Title makes 'improvement' claim which is not substantiated by data
presented in paper - title could be changed to reflect emphasis on
'steps', description of NL_MAMS and its rationale (see comments
above). Abstract rather brief - seems incomplete - it ends '..and
analyze the results of' ???
8. Does the paper adequately refer to related work? Are the references
complete and necessary?
To the best of my knowledge.
9. Overall, is the paper acceptable? Please rate on a scale 1 - 9, as follows:
9 - accept as it is
7 - accept with minor revisions
5 - needs major revision before it could be accepted
3 - needs major revision but may not be acceptable even then
1 - not acceptable
I would assess this paper as 4 in its present form.
----------Review 3------------------------------------------------------------
IJAIED Reviewer Form
--------------------
Paper No: 406
Comments (which will be sent directly to the author)
--------
1. Is the subject of the paper suitable for IJAIED? Yes. The paper
describes the "mental state simulator", an AI program which uses formal
reasoning to teach children diagnosed with high functioning autism about
mental state concepts and representations. The application is novel and
the results with a limited groups of children are promising.
2. Is the content of the paper likely to be of interest to and
appropriate for IJAIED readers? definitely yes. The paper is an
original application of AI formal reasoning to the exciting area of autism
education/therapy.
3. Is the paper technically sound and accurate in its AI and Education
content? to some extent. The authors present a novel approach to
teaching children with HFA about mental states at different levels (i.e.
simple and complex). However, the paper only addresses a limited range of
mental states, and it is hard to immediately see how it would generalize
to other types of mental states (e.g. cognitive states / affective
states). Also, it seems quite theoretic and the children may have
difficulties in the ability to generalise from it to everyday life (unless
the expectation is for children to run these decision trees in their
heads). The paper would benefit from having such a discussion.
4. Is this a new and original contribution? Does the author make clear
what this contribution is? The mental state simulator is a novel
method for teaching children with autism how to understand mental
concepts. The description of the contribution has to be distilled from the
paper, and needs to spelt out in a simple and direct way.
5. Are the major claims and conclusions substantiated? Yes
Have the ideas or systems been tested or evaluated sensibly? to some
extent.
details: The system has only been tested on two children, so it is hard to
draw conclusions on the usability and generalizability of the system
beyond these two cases. However, it is important to note that it in the
field of autism it is hard to find many cases and conduct trials that
extend over such a long period of time.
6. Is the paper clear, explicit, and well-organised? Is the length
appropriate for the content? Are there any gaps or redundancies?
Perhaps this is the biggest problem with this paper. The paper was very
hard read and to follow. Perhaps a simpler description of the technical
details could be added with each section to describe the big picture /
idea behind that section.
7. Are the title and abstract informative? yes
8. Does the paper adequately refer to related work? Are the references
complete and necessary? More references should be added re: the
theory of mind theory of autism.
9. Overall, is the paper acceptable? Please rate on a scale 1 - 9, as
follows: 5
9 - accept as it is
7 - accept with minor revisions
5 - needs major revision before it could be accepted
3 - needs major revision but may not be acceptable even then
1 - not acceptable
10. Any other general comments or specific suggestions for the author?
Since the content of the paper is inherently inter-disciplinary, namely
drawing upon formal reasoning in AI, theory of autism, education, I think
the paper would benefit a lot from being re-written so that it is
accessible to people who are not by necessity familiar with all three
research domains.
For instance, with respect to the AI community, more background should be
given about theory of autism (the autism spectrum of conditions), mental
state concepts, and the different classes within those, and other
(education) methods for autism therapy. The author mentions some
"peculiarities" in reasoning of people with autism that are not described
further, so as it stands its hard to conceptualize exactly how is the use
of mental state concepts in autistic people different from normal ones.
To make the paper somewhat accessible to pyschologists, or
professionals/pracitioners who work with children with HFA (which are
really the potential users of the mental state simulator), the technical
details of the paper would benefit from being re-written so that it is
accessible who are perhaps not familiar with formal reasoning. For
example, its not clear what the traditional modal logic-based formal
representation is (page 2).
The correct citation for the mind reading software is :
Baron-Cohen, S., Golan, O., Wheelwright, S., & Hill, J. J. (2004). Mind
Reading: the interactive guide to emotions. London: Jessica Kingsley
Limited
(www.kjp.com <http://www.kjp.com> ).
----------------------------------------------------------------------