José Hernández-Orallo
Professor, Valencian Research Institute for Artificial Intelligence ....................Universitat Politècnica de València Phone: +34963877007 (Ext:73585), Office: DSIC (1F): 236 Address: Camí de Vera 14, E-46022 València, EU. Email: jorallo@upv.es . Senior Research Fellow, Leverhulme Centre for the Future of Intelligence Research Affiliate, Centre for the Study of Existential Risk Address: 16 Mill Lane, Cambridge, UK. Email: jh2135@cam.ac.uk . Fellow, European Association for AIRecent Highlights
- Co-organising the Evaluation Beyond Metrics workshop 2022 @ IJCAI2022.
- Co-organising the AISafety Workshop 2022 @ IJCAI2022, following the previous AISafety Workshops 2019, 2020 and 2021.
- Three papers accepted for AAAI2022: "When AI Difficulty is Easy: The Explanatory Power of Predicting IRT Difficulty", with F. M.-Plumed, D. C-Falcon and C. Monserrat, "How General-Purpose Is a Language Model? Usefulness and Safety with Human Prompters in the Wild" with P A M Casares, B. S. Loe, J. Burden and S. O'hEigeartaigh, and a Senior Member Track Paper: "Training on the Test Set: Mapping the System-Problem Space in AI", with W. Schellaert and F. M.-Plumed (Blue Sky Idea Runner-Up Award)
- Our paper on "Automating Data Science" with T. De Bie, L. De Raedt, H. H. Hoos, P. Smyth and C. K. I. Williams on the cover of the Communications of the ACM!
- Hernandez-Orallo, J.; Loe, B.S.; Cheke, L.; Martinez-Plumed, F., O h'Eigeartaigh, S. "General Intelligence Disentangled: The Generality of Natural and Artificial Intelligence", Nature Sci Rep 2021.
- New chapter: "Identifying artificial intelligence capabilities: What and how to test", in "AI and the Future of Skills, Volume 1: Capabilities and Assessments", OECD Publishing, Paris.
- Co-organising the SafeAI Workshops at AAAI 2022 following the previous editions in 2019, 2020 and 2021.
- Special Issue Editor on Automating Data Science for the Machine Learning Journal
- New NeurIPS2021 paper: "Think Big, Teach Small: Do Language Models Distil Occam's Razor?" with G. Jaimovich, D.C. Falco and C. Ferri, chapter "Teaching and Explanation: Aligning Priors between Machines and Humans" with C.Ferri in Muggleton, S. and Chater, N. (Eds.) (2021) Human-Like Machine Intelligence. Oxford University Press, and ECML/PKDD 2021 paper "Optimal Teaching Curricula with Compositional Simplicity Priors" with Manual Garcia-Piqueras.
- Participated in a panel at the NIST AI Measurement and Evaluation Workshop in June 2021 and the AI Metrology series in September 2021.
- Co-organised ECML/PKDD Workshop on Automating Data Science 2021.
- New paper accepted the Journal of Artificial Intelligence Research ("Measuring the occupational impact of AI: tasks, cognitive abilities and AI benchmarks" with S. Tolan, A. Pesole, F. Martinez-Plumed, E. Fernandez-Macias and E. Gomez), 2021.
- Co-organised AISafety Workshop 2021 @ IJCAI2021, following the previous AISafety Workshops 2019 and 2020.
- New papers accepted for Artificial Intelligence Journal ("Making sense of sensory input" with Richard Evans et al.), Machine Learning Journal ("AUTOMAT[R]IX: learning simple matrix pipelines"), J. of Intelligent Systems ("Missing the missing values: The ugly duckling of fairness in machine learning"), Nature Mat Intell ("Research Community Dynamics behind Popular AI Benchmarks") (see some coverage here) and Telematics and Informatics ("Futures of artificial intelligence through technology readiness levels") 2021.
- Invited for the Spanish Senate's Commission on Economic Affairs and Digital Transformation, March 2021.
- Paper "Negative Side Effects and AI Agent Indicators: Experiments in SafeLife" at SafeAI Workshop at AAAI 2021.
- Participated in the OECD Expert Meeting on Skills and Tests for Assessing AI and Robotics, with this presentation.
- New papers accepted for Minds and Machines ("Twenty Years Beyond the Turing Test: Moving Beyond the Human Judges Too") and Expert Systems and Applications ("Learning alternative ways of performing a task").
- Animal AI Olympics Paper: "The Animal-AI Testbed and Competition", Proceedings of Machine Learning Research, 2020.
- Co-organised the 1st Workshop on Evaluating Progress in AI (EPAI2020) at ECAI 2020.
- Four papers accepted for ECAI 2020: "Tracking AI: The Capability is (Not) Near" with F. Martínez-Plumed and E. Gómez, "AI Paradigms and AI Safety: Mapping Artefacts and Techniques to Safety Issues" with F. Martínez-Plumed, Shahar Avin, Jess Whittlestone and Seán O h'Eigeartaigh, "Finite and Confident Teaching in Expectation:Sampling from Infinite Concept Classes" with J.A. Telle and "Family and Prejudice: A Behavioural Taxonomy of Machine Learning Techniques" with the DMIP team.
- Read our paper: CRISP-DM Twenty Years Later: From Data Mining Processes to Data Science Trajectories, IEEE Transactions on Knowledge and Data Engineering journal, 2020.
- Read our paper: "Does AI Qualify for the Job? A Bidirectional Model Mapping Labour and AI Intensities", AAAI/ACM Conference on Artificial Intelligence, Ethics, and Society, 2020.
- Co-organised the Animal-AI Olympics, 2019.
- Read: Journal of Artificial Intelligence Research: "AI Generality and Spearman's Law of Diminishing Returns", 2019.
- Contributing to the AI Safety Landscape.
- Gave a talk at the Cambridge Science Festival, 2019.
- Measure for measure column: "Unbridled mental power", Nature Physics, vol. 15, 2019.
- Read our paper Item Response Theory in AI: Analysing Machine Learning Classifiers at the Instance Level", Artificial Intelligence Journal, 2019.
RESEARCH
One of the great scientific challenges of this century is to understand what intelligence is and how it can be recreated. My bit is, on one hand, the evaluation and measurement of intelligent systems in general and machine learning in particular and, on the other hand, some more applied research on data science, data mining and inductive programming. However, I'm interested in many other things, and my publication profiles below can give a better account of what my research really looks like:- Google Scholar Profile.
- Publication list (with preprints). Not always up-to-date.
- Identifying Artificial Intelligence Capabilities: What and How to Test, OECD Expert Meeting on Skills and Tests for Assessing AI and Robotics, 5-6 October 2020.
- Measuring Intelligence Incrementally: Search, Demonstration and Transmission, DARPA Machine Common Sense PI Meeting, July 21-23, 2020.
- AI Paradigms and AI Safety: Mapping Artefacts and Techniques to Safety Issues, European Conference on Artificial Intelligence, Aug 29 - Sep 8, 2020.
- On Broken Yardsticks and Measurement Scales, (paper), MetaEval@AAAI2020, New York, Feb 8, 2020.
- Surveying Safety-Relevant AI Characteristics, SafeAI2019 AAAI Workshop, Honolulu, 27 January 2019.
- AI Extenders: The Ethical and Societal Implications of Humans Cognitively Extended by AI, AAAI / ACM Conference on Artificial Intelligence, Ethics and Society, Honolulu, 27-28 January 2019.
- Artificial Intelligence : Present, Future and Beyond, [video], HUMAINT Winter school on AI and its ethical, social, legal and economic impact, JRC Seville, February 4-8th, 2019.
- The What and How of AI Evaluation, [video], HUMAINT Winter school on AI and its ethical, social, legal and economic impact, JRC Seville, February 4-8th, 2019.
- Artificial Intelligence and Data Science : Looking Ahead, Springer Nature Conference on Artificial Intelligence in Biomedicine, Madrid 7th February 2019.
- Diversity Unites Intelligence: Measuring Generality at Varieties of Minds 2018 in Cambridge, 2018
- Measuring A(G)I Right: Some Theoretical and Practical Considerations at DeepMind, London, 2018
- Natural and Artificial Intelligence: Measures, Maps and Taxonomies at Clare Hall, Cambridge, 2018
- The Mythical Human-Level Machine Intelligence at Philosophy and Theory of AI in Leeds, 2017
- FHI, Oxford, Aug 2016, or the very similar CSER, Cambridge, Aug 2016, about the measurement of natural and artificial intelligence
- Machine learning performance evaluation: tips and pitfalls at PAPIs Connect, 2016
- AGI 2015 Tutorial: Evaluation of Intelligent Systems
- Invited Talk at ECML Workshop on Learning over Multiple Contexts 2014
- Machine intelligence evaluation: from the Turing Test to the present Day (and beyond), for a Turing centenary celebration
TEAMS
Data Mining, Machine Intelligence and Inductive Programming (DMIP), part of the ELP group.- You can see the projects, software and other activities of our group here.
- If you're a student or an academic and want to collaborate with us or get involved with our group, please feel free to contact us.
- We are part of the Valencian Research Institute for Artificial Intelligence
- I participate in this programme and others at the CFI. Stay tuned!
TEACHING
- More than 15,000 students (as for 2020) on my MOOC EDX course on Machine Learning and Data Science (in Spanish) I currently teach several courses on data science and data mining at the master level.
- I also had an undergraduate database course for many years.
TRANSFER
We have had projects, collaborations and visits with several companies in different areas: health, retailing, software development, automotive, ... Recently, I've been managing two "Cátedras/Aulas de Empresa":- Agreement (cátedra) with BigML on machine learning.
- Agreement (aula) with Everis on employability in information technology.
SHORT BIO
José Hernández-Orallo is Professor at the Universitat Politècnica de València, Spain and Senior Research Fellow at the Leverhulme Centre for the Future of Intelligence, University of Cambridge, UK. He received a B.Sc. and a M.Sc. in Computer Science from UPV, partly completed at the École Nationale Supérieure de l'Électronique et de ses Applications (France), and a Ph.D. in Logic and Philosophy of Science with a doctoral extraordinary prize from the University of Valencia. His academic and research activities have spanned several areas of artificial intelligence, machine learning, data science and intelligence measurement, with a focus on a more insightful analysis of the capabilities, generality, progress, impact and risks of artificial intelligence. He has published five books and more than two hundred journal articles and conference papers on these topics. His research in the area of machine intelligence evaluation has been covered by several popular outlets, such as The Economist, New Scientist or Nature. He keeps exploring a more integrated view of the evaluation of natural and artificial intelligence, as vindicated in his book "The Measure of All Minds" (Cambridge University Press, 2017, PROSE Award 2018). He is a member of AAAI, CLAIRE and ELLIS, and a EurAI Fellow.IN THE MEDIA (and blogs)
Don't take this too seriously:- The Economist, March 3rd 2011
- New Scientist Sat 10/Sept/2011, pp42-45.
- Alpha Galileo. World's independent source of research news
- Eurekalert. Global news service of the AAAS (American Association for the Advancement of Science)
- ACM Tech News
- Vokrug Sveta (Russia)
- BBC News
- Nature
- Inverse
- Financial Times
- ...