constrðX=Condition >ConstraintÞ:(2) The system will apply the Constraint to all and only the X for which Condition is true. Notice that, to the best of our knowledge, the notation proposed in Eq. (2) is new w.r.t. the application domain, but is rather similar to many other notations used to apply constraint on a restricted set of elements, selected from a larger set through the specification of a condition.The top level solver declares a generic predicate solveðþProblem;þImg; ResultÞ(3) where Problem contains the puzzle’s description through the Prolog term Category(Parameter1, Parameter2, . . .) presented before, Img is the OpenCV object representing the image, and Result may be unified with a word, a number, or a modified version of the input Img with the solution drawn on it. The actual implementation of the solve predicate is assigned to the category-specific solvers, which are examined in the following.Numbers diagram resolutionIn the diagram solver, the puzzle’s category and definition can be stated through the Prolog following term: numbersdiagramðConstraintsList;ResultÞ:(4) where the first parameter (ConstraintsList) is a list of constraints on the puzzle’s variables and Result is another constr term that the solver will unify with the solution of the problem. The constraints in the ConstraintsList can be expressed by either CLP(FD) or other predicates defined by the library for convenience’s sake.For example, the puzzle of Figure 2d can be described by the following logic form. numbersdiagramð½constrðI=ðvariableðIÞ;varshapeðI;circleÞÞ >ðvarvalueðI;XÞ;Xin3::8ÞÞ;constrðL=ðlineðLÞ;lengthðL;3ÞÞ >ðlinevalsðL;LXÞ;sumðLX;#¼;18ÞÞÞ�;constrðR;ðvarsbypositionð½bottom;left�;½Ij�Þ;varvalueðI;RÞÞÞÞ:(5) The first constr term in the ConstraintsList states that “For any variable I such that it is associated to a circle shape in the figure, its value must be constrained in ½3;8�“. The second constr term expresses the requirement that the sum of the numbers on the same line of boxes must be 18. To state this constraint, the line and line_vals predicates are employed to address the aligned boxes and the values inside them, respectively. Finally, the last constr term is related to the result R and specifies we want to extract the value of the variable located in the bottom left box.The term in Eq. (5) is passed as first parameter to the solve predicate of Eq. (3) in order to trigger the image analysis and the resolution process. In particular, for numbers-in-diagram puzzles, the goal is to insert numbers in the boxes of the diagram according to a certain logic. Therefore the image analysis must first focus on extracting information about the presence and disposition of the boxes in the diagram. This information is represented in the form diag(BoxList, LineList). As the variable’s name suggest, BoxList is a list of boxes, each represented by a specific box(Poly, Colour, Txt) predicate, where the parameters clearly report the type of polygon realising the box, its colour, and its possibly enclosed text. LineList is the list of spatial relations between the boxes. Such relations are represented through line(LineElements, Orientation) predicates, where Orientation specifies the horizontal, vertical, or oblique spatial relation between the set of boxes in LineElements.Whereas these definitions have a rather simple semantics, from the image processing point of view, the identification of what is a box, and the process of distinguish it from other closed forms in the diagram is less straightforward. The human brain usually classifies as boxes all those closed spaces corresponding to geometric figures or characterised by relevant symmetries. For this reason, 12R. BUSCAROLI ET AL.
we currently identify circles, rectangles and equilateral polygons as boxes, but this criterion can be extended by declaring additional shapes accepted as boxes. This operation can be easily performed by changing of the true_box(Box) predicate .The search of spatial relations between the boxes have similar issues. In some diagrams, adjacent boxes are polygons with common borders or vertex as in Figure 2c. In some others, the spatial relation is defined by straight, curve, continuous or dashed lines connecting the boxes as shown in Figure 2d. Two different kinds of image analysis are therefore needed. To this end, the library first identifies relations of the first type (polygons with common vertex and borders), then considers all the segments that do not belong to any box and performs the second type of search (boxes connected by segments).Once the image features are extracted, the CSP model of the problem must be defined. As these puzzles require writing numbers inside the boxes, a variable is associated to each box. If the box already contains a number in the original image, the variable is obviously associated with such number.The number recognition is the most expensive task of the diagram solving process. When the problem does not actually require to recognise numbers (as it is, e.g. in the puzzle of Figure 1), that task can be skipped by employing the term diagram(ConstraintsList,Result). The puzzle in Figure 1 can be formulated as follows. diagramð½constrðI=variableðIÞ >ðvarvalueðI;XÞ;countvarsðNÞ;Xin1::NÞÞ;constrððI1;I2Þ=ðvariableðI1Þ;variableðI2Þ;I1¼I2;adjacentðI1;I2ÞÞ >ðvarvalueðI1;X1Þ;varvalueðI2;X2Þ;X1#¼X2ÞÞ;constrðminimizemaxÞ�;constrðR;ðvalsðLXÞ;maxlistðLX;RÞÞÞÞ:where the first constr term states that any variable I associated with a box must be an integer in the range between 1 and the total number of employed variables; the second constr term enforce to assign different values to those variables that are associated with adjacent boxes; and the third constr term minimises the total number of employed variables. Finally, the Result parameter is again a constr term which states to return the maximum value associated to a variable in the solution. The problem can be formulated in the same way by using numbers_diagram instead of diagram, but in that case the image processing task would be triggered before any reasoning to look for numbers in the boxes (in vain).Geometrical figures resolutionWhen the puzzle requires to identify and enumerate certain shapes in the diagram, the polygon solver must be called. In this solver, the problem type can be specified by two predicates.. countpolygonsðShapeÞ(6) countpolygonsðShape;constrðX;FilterÞÞ(7) where constr(X,Filter) allows to specify if some additional constraint must be fulfilled besides the congruence with the specified Shape. For example, we could require the polygon to be equilateral or have a certain area (as in Figure 2a). Since in these puzzles the area is usually defined in terms of the number of base polygons composing a grid (e.g. in Figure 2a we look for squares composed of 4 pieces), a min area(-Area) predicate is defined to extract the area of the smaller polygon in figure (i.e., the piece). This polygon can be of any kind, not necessarily a square. Another predicate, grid area(+Poly,-N), unifies the N variable with the number of minimal polygons needed to cover the area of the input polygon Poly. As an example, the logic programming description of Figure 2a is simply.. JOURNAL OF EXPERIMENTAL & THEORETICAL ARTIFICIAL INTELLIGENCE13
countpolygonsðsquare;constrðX;gridareaðX;4ÞÞÞ:which instructs the solver to identify and count the number of squares whose area is equal to 4 grid elements.Despite the short logic-oriented description of the problem, the shape recognition task under-neath is not trivial. Instead of looking for all possible polygons in figure, and discard the ones not fulfiling the requirements – a strategy which can be time consuming – our algorithm starts from an image segment, and compose a polygonal chain by progressively adding a segment at a time until the desired number is reached. A line cannot be added if it intersects the chain in a point different from the initial one, and if two segments are aligned, they are obviously considered as a single segment. If the required number of sides is reached without closing the chain, the algorithm backtracks on other image segments. In this way, no time is wasted on searching polygons with more edges than required.Grid resolutionIn case of spatial logic problems, such as tiling puzzles, or games where a grid must be partitioned into a certain number of equal pieces, the grid solver module is employed. Similarly to the polygon solver, for this solver the puzzle’s logic description is again rather synthetic. It can be represented by two alternative predicates: puzzleðOptionsÞ(8) dividesameshapeðN;OptionsÞ(9) where the puzzle predicate is used to address tiling puzzles (where a set of given pieces must be put on a grid); whereas divide_same_shape is devoted to situations where only the grid is provided in the figure, and the text requires to divide it into a certain number of equal pieces. The Options parameter is provided because often the puzzle allows to perform spatial operations on the pieces, such as rotate or overturn. The parameter N of divide_same_shape instead expresses the number of pieces in which the grid should be divided.As an example, the puzzle in Figure 2b can be described by the following. dividesameshapeð2;½rotate;overturn�Þ:In case of puzzle(Options) problems, the diagram reports both the grid and the pieces to be scattered on it. So, all the external borders of the picture’s elements are considered and the bigger one is found to be the grid (which is reasonable because it has to contain all the others). For both puzzle and divide same shape, the grid analysis starts by identifying its shape in terms of all its minimal building blocks: the smallest square of the grid, which the solver will represent through the position of its upper left vertex. The grid is therefore rendered as if it was on a Cartesian plane: a predicate grid_diag(LX,LY,Grid) serve the purpose. LX and LY are the lists of axis coordinates corre-sponding to the upper left vertices, and Grid is a matrix (a list of lists with the same length) where each element is either 0 or −1, to signify that a square was found or not found on the corresponding coordinates. Indeed grids might not always be convex forms, such as squares or rectangles. The strategy of representing the grid through two lists of coordinates and a matrix of 0/-1 allows to easily represent grids with various forms.The solving strategy searches the first uncovered grid’s square and tries to cover the connected space around it with one of the pieces, possibly rotating and overturning it if such options are allowed. If the solver cannot cover the connected space, it backtracks by reconsidering the latest added piece. In case of divide same shape, the algorithm does not know the shape of the pieces, but only that their area must be the total area of the grid divided by N. So, the algorithm makes a first attempt to define a possible shape and generates N identical pieces to occupy the grid.14R. BUSCAROLI ET AL.
Since for a machine the challenges in solving one of these puzzles is clearly not in the dimension of the problem, we do not investigate the performance of the library for increasing number of variables and constraints. However, we underline that the most time-consuming task is text detec-tion and recognition, whereas the executions of the grid, polygon or diagram solvers (once the numbers are recognised) all require a very limited amount of time, in the order of milliseconds. This was indeed expected, because the dimensions of the considered problems are extremely small with respect to the computing capabilities of modern hardware.The whole source code of our spatial reasoning library can be found on GitHub5.Experimental evaluationIn our framework, the puzzle resolution process involves several different tasks, from the extraction/ definition of the logic-based description of the puzzle, to the resolution of the problem specified in a computer-understandable form. The latter in particular is of less interest: as we discussed in Section 1, in the AI research field many successes have been obtained when tackling with specific problems. With this respect, the mathematical puzzles are indeed toy problems w.r.t. the currently available resolution techniques. For example, let us consider the numbers-in-diagram category games: when properly mapped, the puzzles become CSP problems of some ten variables and twenty constraints at most (remember that puzzles are aimed to human recreation). The performance of Constraint Programming solvers is usually measured against problem instances with tenth of thousands of variables, and hundred thousand constraints. As a consequence, the computational time required to solve a properly represented puzzle is usually in the order of nanoseconds, thus making any measurement meaningless. For this reason, in subsection 5.1 we briefly discuss the computational performances of the solving process from the final user perspective, by simply exploiting the demo website.10Of the greatest interest instead is the evaluation of the quality of the solving process. As previously discussed, the ‘hard’ part of solving the puzzles is getting a correct computer-under-standable representation of the puzzle. In subsection 5.2 we discuss a more qualitative evaluation of our approach, trying to highlight achievements but also weaknesses of the current framework. We do not introduce any metric on the quality of the obtained puzzle’s model, but rather we investigate if and when our framework manage to achieve a puzzle model that will in turn lead to a correct solution.The demo website and its performanceA demo of our framework is available online10, and interested users can freely experiment with part of the presented tools.As shown in Figure 5, the web-based application presents four areas: the ‘Human readable’ area is devoted to show the puzzle in its original aspect, while the ‘Prolog translation’ area shows the prolog query whose parameters are indeed the problem description in logical terms. The user is also invited to freely change the parameters and experiment with the reasoning library: tips are provided suggesting parameters that indeed will make the puzzle feasible, thus leading to a solution.The user can freely choose among fifteen puzzles: upon the selection of a puzzle, its image and textual description are shown, together with the problem translation into a Prolog data structure. By pressing the button ‘Solve’, the selected image with its text, and the Prolog query, are processed to the server, which start a Prolog interpreter.11 The ‘SWIPL Console’ area reports the output of the Prolog interpreter running on the server, while the‘’Output image” shows those solutions that are required to be drawn on top of the original image (such as, e.g. the geometrical puzzles). At every pressing of the ‘Solve’ button the time to execute the SWIPL invocation is reported below the ‘SWIPL Console’ area.JOURNAL OF EXPERIMENTAL & THEORETICAL ARTIFICIAL INTELLIGENCE15
In Table 1 we report the execution time (in sec.) requested to solve each of the puzzles available in the demo web application. The reported times are not comprehensive of the network overload. For each puzzle, resolution times are taken for ten consecutive tests, and the average time is reported.Figure 5. Web interface of the demo application.Table 1. Execution times of the demo web application. Average over ten runs for each puzzle.Puzzle #TypeTime (sec.)1Spatial Logic0.7132Numbers-in-diagram4.6723Geometrical figure0.4914Numbers-in-diagram0.8305Spatial Logic0.6536Numbers-in-diagram11.2757Numbers-in-diagram19.5188Numbers-in-diagram31.9689Spatial Logic0.50310Spatial Logic0.50411Numbers-in-diagram17.71012Geometrical figure0.54613Spatial Logic0.50014Numbers-in-diagram25.40115Spatial Logic0.51716R. BUSCAROLI ET AL.
The demo web application currently suffers some limits, partly because of security reasons, and partly as a consequence of some limits of our framework (see, Section 6 for a discussion): (i) it is not possible to upload a puzzle: the user can experiment only with the proposed fifteen examples; (ii) the puzzles are already classified into the right problem type; (iii) the problem description in logical terms is already provided, instead of being generated on the fly: the user can however modify it before starting the reasoning.Qualitative assessment of the approachSeveral different aspects have a great influence on the quality of the system outcomes. Regarding the diagram-related part, there is of course the quality of the image: jpeg artefacts or a too much low resolution might lead the OpenCV library to detect false edges, or to miss existing ones. For example, the recognition of polygons is based on the assumption that edges indeed meet each other in some vertex: the presence or absence of some pixel around such vertex might induce the library to false conclusions. Even, the font adopted for indicating some number directly in the diagram might impact the performances of the OCR component. Scanned images in particular might present distortions of the characters, thus leading to recognition mistakes.Regarding the text-related part, we might stress that the variability in describing the puzzles heavily affects the data extraction module. Such problem is exacerbated from the fact that the available dataset is in Italian language only. We worked on an English-translation of the problems, but we can’t exclude that we involuntary introduced some language bias during the translation step. Resorting to translation software lead to poor results, given that the puzzle’s texts are usually very short sentences. Alternatively, we investigated the adoption of NLP tools for the Italian language, but when confronted with analogous tools for the English language, the maturity level of the former appears to be lower. A further difficulty relates to the length of the puzzle’s texts, combined with (sometime useless) references to a real-life context. While such references help the human to better understand the problem, they might result as misguiding information when dealing with NLP tools.The use of Deep Learning-based approaches might appear as the perfect answer to the problems highlighted above. However, the total number of available mathematical puzzles counts up to around a hundred of examples, a number far insufficient for any Machine Learning training algo-rithm. Fine tuning of pre-trained neural networks might not be feasible as well, given the small dimension of the training data.ConclusionMaths puzzles with diagrams, even the most simple, are a perfect training field for AI. Indeed, despite the impressive progresses made so far, a lot of work is still needed to reach the ambitious goal of a machine autonomously solving a maths puzzle from its original textual and graphical formulation. No sub-symbolical technique can reach this goal alone. The case study of maths puzzles with diagrams inevitably requires a multimodal approach to problem solving, which combines sub- symbolical techniques to extract relevant information from text and diagram into a unique frame-work. We claim that such task must be guided by reasoning. Logic and its symbolical approach are therefore essential in this scenario.Focusing on maths puzzles with diagrams, we propose a framework to allow knowledge integra-tion of concepts expressed by the text with what can be extracted from the diagram. In particular, we propose a Data Extraction module, which follows a decision tree of questions from the most general to the most specific in order to deepen the knowledge about the puzzle’s request; and a set of primitives, which interact with image processing software to extract and reason upon spatial information about the elements in the figure. Similarly to the human cognitive skills needed to reason on maths puzzles, the functionalities offered by the proposed software are organised into overlapping layers, from the basic to the most complex ones.JOURNAL OF EXPERIMENTAL & THEORETICAL ARTIFICIAL INTELLIGENCE17
The result is interesting not only for the wide set of puzzles that can be solved with our library but, more notably, because allows to verify that logic programming is the right tool to manage the complexity of the solving process, even when a multimodal approach is needed. Indeed, thanks to Prolog and the CLP(FD) library, we can easily express the knowledge extracted from the image and combine it with a logical description of the natural language text, leaving the burden of finding the best solving strategy for the puzzle to the underlying theorem prover.Furthermore, we believe that the proposed decision tree of questions useful to data extraction represents itself a relevant contribution of our work, because it can be used by human beings to support the resolution process. For example, it can represent a guide to solve the puzzle for those children experiencing some difficulties in the identification of the correct resolution method; or it can have a didactical value if followed by the children looking for the relevant data in the text – thus helping the teacher to explain the crucial points of the problem description.Our work also helps to highlight the limits of current approaches and the next steps required. As already discussed, the image processing and OCR software may show some shortcomings when recognising distorted letters and numbers. Besides that, an additional effort is needed to enrich the concepts that the library can manage. For example, the current version of the software can recognise geometrical figures, numbers, letters, and some more general concepts, like for example, the notion of ‘box’, which in the puzzle’s context can be a shape with a variety of forms. However, if the puzzle required to identify different real-world concepts (e.g. arrows, card’s suits, etc.), the integration of other, more sophisticated image recognition techniques into the library would be required.The spatial reasoning library can manage puzzles that falls into the categories of numbers-in- diagram, geometrical figures, and spatial reasoning. These classes together cover the majority of puzzles from the considered source1, but another important part is represented by crypto-arithmetic games. This category could be integrated with a relatively contained effort because most of the spatial reasoning tools required for solving such puzzles is already provided by the library.In our opinion, the biggest limit of our approach (and also the toughest problem so far) is related to the integration of NLP techniques to properly recognise the puzzle’s type and to convert the problem’s textual description into a set of logic facts and clauses. As mentioned earlier, we investigated different techniques, but none of them provided good results and at the same time proved to be robust enough against unforeseen examples. Other solutions might be inspired by some previous works on this topic (Berger et al., 1996; Mitra & Baral, 2015): we plan to investigate these techniques as well in the near future.Finally, in the long run a challenging task would be to widen the application scope to contexts other than maths puzzles, where a problem-solving-oriented reasoning is required. An ambitious example of future work is the automated reasoning on technical problems after having understood the content of a related manual (with natural language descriptions and graphical schemas) without human intervention. In this regard, our proposal to integrate sub-symbolical techniques to account for perceptive tasks (like image processing) and symbolical techniques for higher-level reasoning reflects a widespread trend in model AI. In the broad range of scenarios that can benefit from a simplified human–machine interaction, the lesson learned from the multimodal approach to puzzle resolution can be extremely useful.Notes1. Catalogue of maths puzzles – PRISTEM Research Centre, Bocconi University of Milan. http://matematica.uniboc coni.it/articoli/archivio-giochi2. Among the so called ‘recreational games’, there are also puzzles whose diagram and text purposely try to fool the user into classifying it in the wrong problem type. In this work we do not consider such a class of puzzles.3. https://stanfordnlp.github.io/CoreNLP/4. https://wordnet.princeton.edu/5. https://github.com/ai-unibo/spatial-reasoning6. https://opencv.org18R. BUSCAROLI ET AL.
7. https://opensource.google/projects/tesseract8. https://www.swi-prolog.org/pldoc/man?section=cpp-overview9. https://www.swi-prolog.org/man/clpfd.html10. http://games-ai.disi.unibo.it/games11. We choose to rely on the SWI-Prolog Interpreter https://www.swi-prolog.org/AcknowledgementsThis work has been partially supported by the European Union’s H2020 projects AI4EU (g.a. 825619) and TAILOR (g.a. 952215).Disclosure statementNo potential conflict of interest was reported by the author(s).FundingThis work was supported by the Horizon 2020 Framework Programme [825619,952215].ORCIDFederico Chesani http://orcid.org/0000-0003-1664-9632Daniela Loreti http://orcid.org/0000-0002-6507-7565Paola Mello http://orcid.org/0000-0002-5929-8193ReferencesAlzboon, L., & Nagy, B. (2020). Truth-teller-liar puzzles with self-reference. Mathematics, 8(2), 190. https://doi.org/10. 3390/math8020190 Andreas, J., Rohrbach, M., Darrell, T., & Klein, D. (2015). Deep compositional question answering with neural module networks. CoRR, abs/1511.02799 doi:10.48550/arXiv.1511.02799 .Antol, S., Agrawal, A., Lu, J., Mitchell, M., Batra, D., Zitnick, C. L., & Parikh, D. (2015). VQA: Visual question answering. In 2015 IEEE International Conference on Computer Vision (ICCV), Santiago, Chile, (pp. 2425–2433). IEEE Computer Society.Baral, C. (2009). Knowledge representation, reasoning and declarative problem solving. Cambridge University Press.Berger, A. L., Pietra, S. D., & Pietra, V. J. D. (1996). A maximum entropy approach to natural language processing. Computational Linguistics, 22(1), 39–71 https://aclanthology.org/J96-1002.pdf .Bhatt, M., & Flanagan, G. (2010). Spatio-temporal abduction for scenario and narrative completion. Proc. of the int. workshop on spatio-temporal dynamics, Lisbon, Portugal (p. 31–36). IOS Press.Campbell, M., Jr., Hoane, A. J. H., & Hsu, F. (2002). Deep blue. Artificial Intelligence, 134(1–2), 57–83. https://doi.org/10. 1016/S0004-3702(01)00129-1 Chesani, F., Mello, P., & Milano, M. (2017). Solving mathematical puzzles: A challenging competition for AI. AI Magazine, 38(3), 83–96. https://doi.org/10.1609/aimag.v38i3.2736 Cheung, B. (2016a). Spatial reasoning explained. http://bennycheung.github.io/spatial-reasoning-explained Cheung, B. (2016b). Using prolog to solve logic puzzles. http://bennycheung.github.io/using-prolog-to-solve-logic- puzzles Csenki, A. (2006). Enigma 1225: Prolog-assisted solution of a puzzle using discrete mathematics. Computers & Mathematics with Applications, 52(3–4), 383–400. https://doi.org/10.1016/j.camwa.2006.03.020 De Raedt, L., Frasconi, P., Kersting, K., & Muggleton, S. (Eds.). (2008). Probabilistic inductive logic programming - theory and applications (Vol. 4911). Springer.De Raedt, L., Kersting, K., Natarajan, S., & Poole, D. (2016). Statistical relational artificial intelligence: Logic, probability, and computation. Morgan & Claypool Publishers.De Raedt, L., & Kimmig, A. (2015). Probabilistic (logic) programming concepts. Machine Learning, 100(1), 5–47. https:// doi.org/10.1007/s10994-015-5494-z JOURNAL OF EXPERIMENTAL & THEORETICAL ARTIFICIAL INTELLIGENCE19
Devlin, J., Chang, M., Lee, K., & Toutanova, K. (2019). BERT: Pre-training of deep bidirectional transformers for language understanding Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) Minneapolis, Minnesota (pp. 4171–4186) Association for Computational LinguisticsDries, A., Kimmig, A., Davis, J., Belle, V., & De Raedt, L. (2017). Solving probability problems in natural language. Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence (IJCAI-17), Melbourne, Australia, 3981–3987.Forbus, K. D., Klenk, M., & Hinrichs, T. R. (2009). Companion cognitive systems: Design goals and lessons learned so far. IEEE Intelligent Systems, 24(4), 36–46. https://doi.org/10.1109/MIS.2009.71 Fukui, A., Park, D. H., Yang, D., Rohrbach, A., Darrell, T., & Rohrbach, M. (2016). Multimodal compact bilinear pooling for visual question answering and visual grounding. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, Austin, Texas, (pp. 457–468). The Association for Computational Linguistics.Gambetti, E., Buscaroli, R., Chesani, F., Giusberti, F., Loreti, D., & Mello, P. (2020). Artificial intelligence and cognitive psychology: How to solve mathematical problems. Sistemi Intelligenti, 32(2), 287–316 doi:10.1422/96329.Gelfond, M., & Lifschitz, V. (1991). Classical negation in logic programs and disjunctive databases. New Generation Computing, 9(3–4), 365–386. https://doi.org/10.1007/BF03037169 Hochreiter, S., & Schmidhuber, J. (1997). Long short-term memory. Neural Computation, 9(8), 1735–1780. https://doi.org/ 10.1162/neco.1997.9.8.1735 Hosseini, M. J., Hajishirzi, H., Etzioni, O., & Kushman, N. (2014). Learning to solve arithmetic word problems with verb categorization. Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), Doha, Qatar (pp. 523–533). Association for Computational Linguistics.Jaffar, J., & Maher, M. J. (1994). Constraint logic programming: A survey. The Journal of Logic Programming, 19-20(20), 503–581. https://doi.org/10.1016/0743-1066(94)90033-7 Kembhavi, A., Seo, M. J., Schwenk, D., Choi, J., Farhadi, A., & Hajishirzi, H. (2017). Are you smarter than a sixth grader? Textbook question answering for multimodal machine comprehension. In 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Honolulu, HI, USA (pp. 5376–5384). IEEE Computer Society.Koller, D., Friedman, N., Džeroski, S., Sutton, C., McCallum, A., & Pfeffer, A., others. (2007). Introduction to statistical relational learning. MIT press.Krishnamurthy, J., Dasigi, P., & Gardner, M. (2017). Neural semantic parsing with type constraints for semi-structured tables. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing (pp. 1516–1526). Association for Computational Linguistics.Lev, I., MacCartney, B., Manning, C. D., & Levy, R. (2004). Solving logic puzzles: From robust processing to precise semantics. Proc. of the 2nd workshop on text meaning and interpretation (pp. 9–16). Association for Computational Linguistics.Liang, C., Hsu, K., Huang, C., Li, C., Miao, S., & Su, K. (2016). A tag-based English math word problem solver with understanding, reasoning and explanation. In Proceedings of the 2016 conference of the North American chapter of the association for computational linguistics: Demonstrations (pp. 67–71). The Association for Computational Linguistics.Loreti, D., Chesani, F., Mello, P., Roffia, L., Antoniazzi, F., Cinotti, T. S., . . . Costanzo, A. (2019). Complex reactive event processing for assisted living: The habitat project case study. Expert Systems with Applications, 126, 200–217. https:// doi.org/10.1016/j.eswa.2019.02.025 Lu, J., Yang, J., Batra, D., & Parikh, D. (2016). Hierarchical question-image co-attention for visual question answering. In D. Lee, M. Sugiyama, U. Luxburg, I. Guyon, & R. Garnett (Eds.), Advances in Neural Information Processing Systems (Vol. 29, pp. 289–297). Curran Associates, Inc.Manning, C. D., Surdeanu, M., Bauer, J., Finkel, J., Bethard, S. J., & McClosky, D. (2014). The Stanford CoreNLP natural language processing toolkit. Association for Computational Linguistics (ACL) System Demonstrations, Baltimore, Maryland (Association for Computational Linguistics) (pp. 55–60). http://www.aclweb.org/anthology/P/P14/P14- 5010 Mitra, A., & Baral, C. (2015). Learning to automatically solve logic grid puzzles. Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing (pp. 1023–1033). The Association for Computational Linguistics.Nagy, B. (2003). Boolean programming, truth-teller-liar puzzles and related graphs. Proceedings of the 25th international conference on information technology interfaces, 2003. iti 2003, Cavtat, Croatia (IEEE). (p. 663–668).Ohlbach, H., & Schmidt-Schauß, M. (1985). The lion and the unicorn. Journal of Automated Reasoning, 1(3), 327–332. https://doi.org/10.1007/BF00244274 Ramsey, B. D. (1986). The Lion and the Unicorn Met PROLOG. ACM SIGPLAN Notices, 21(8), 62–70. https://doi.org/10. 1145/382278.382395 Russell, S. J., & Norvig, P. (2010). Artificial intelligence - A modern approach, third international edition. Pearson Education.Schneider, W., & McGrew, K. (2012). The Cattell-Horn-Carroll model of intelligence. In Contemporary intellectual assessment: Theories, tests, and issues (3rd ed.). (pp. 99-144). Guilford Press.20R. BUSCAROLI ET AL.
Seo, M. J., Hajishirzi, H., Farhadi, A., Etzioni, O., & Malcolm, C. (2015). Solving geometry problems: Combining text and diagram interpretation. In Proceedings of the 2015 conference on empirical methods in natural language processing (pp. 1466–1476). The Association for Computational Linguistics.Silver, D., Huang, A., Maddison, C., Guez, A., Sifre, L., & van den Driessche, G., & others. (2016). Mastering the game of go with deep neural networks and tree search. Nature, 529(7587), 484–489. https://doi.org/10.1038/nature16961 Smullyan, R. (1978). What is the name of this book? Prentice-Hall, Inc.Zellers, R., Bisk, Y., Farhadi, A., & Choi, Y. (2019). From recognition to cognition: Visual commonsense reasoning. 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (pp. 6720–6731). Computer Vision Foundation/IEEE.JOURNAL OF EXPERIMENTAL & THEORETICAL ARTIFICIAL INTELLIGENCE21