---
title: "Recognising food is not weighing it: Vision LLMs tested"
description: "AI and nutrition: a study in Nutrients tests Vision LLMs on 200 photos of real meals. They recognise the food well; estimating its weight is much harder."
url: "https://www.fastal.it/en/blog/2026-11-ai-nutrizione-benchmark-vision-llm/"
lang: "en"
published: "2026-11-03"
author: "Stefano Capezzone"
tags: ["Artificial Intelligence", "Vision LLM", "Digital Health", "Digital Nutrition", "Benchmark", "Open Source"]
sources: ["https://doi.org/10.3390/nu18213582", "https://github.com/Metabolic-Intelligence-Lab/vlm-food-benchmark", "https://www.metabolicintelligence.org/", "https://www.newswise.com/articles/developed-the-world-s-first-quantum-metabolic-avatar-a-digital-twin-of-human-metabolism-powered-by-quantum-algorithms", "https://byo-me.it/", "https://ollama.com/"]
---

Take a photo of your canteen tray: an artificial intelligence model can tell you what you are eating, but can it also tell you how many grams are on the plate? That is the question answered by a study published on 29 October 2026 in the scientific journal *Nutrients* ([DOI 10.3390/nu18213582](https://doi.org/10.3390/nu18213582)). It puts **Vision-Language Models (VLMs)** — models that read an image and describe its content in natural language — to the test on 200 photographs of real meals, with 870 food items weighed one by one on a scale.

The answer is clear: the best models recognise most of the food, but when it comes to portion size none of them beats a trivial rule, assigning each food the weight of its typical serving. The study, signed by researchers of the Metabolic Intelligence Lab at Università Cattolica del Sacro Cuore and by authors from two companies, including the author of this article, is part of a wider project: an AI-powered platform for precision medicine in nutrition. Fastal provided the infrastructure on which the open-source models were run and consultancy on LLM technologies, and is one of the technology partners of the [Quantum Metabolic Avatar](https://www.newswise.com/articles/developed-the-world-s-first-quantum-metabolic-avatar-a-digital-twin-of-human-metabolism-powered-by-quantum-algorithms) project and of [BYO-Me](https://byo-me.it/), the AI nutrition assistant recently released as an app for Android and iOS.

## At a glance

| Question | Answer | Where in the paper |
|---|---|---|
| What was measured? | Food recognition and portion weight estimation from a single photo | Sec. 2 |
| On what data? | 200 smartphone photos taken by 6 participants, 870 items weighed on a kitchen scale | Sec. 2.1 |
| Which models? | 26 configurations of 17 open-source models, plus 5 proprietary models as a reference | Table 1, Sec. 2.2.5 |
| Best open-source recognition | Qwen2.5-VL 7B, F1 = 0.68 | Table 2 |
| Best open-source portion estimate | Qwen3.5 9B, NMAE = 70 g | Table 2 |
| Best model overall | GPT-4o, F1 = 0.77 and NMAE = 58 g | Table 2 |
| Can the models weigh food? | No better than assigning each item its typical serving (mean error 35 g) | Sec. 3, Table 3 |

## Why recognising and weighing food is hard

Keeping a food diary is tedious, and people who do it tend to under-report what they eat: it is a well-known weakness of self-reporting. Photographing the plate and letting an app fill in the diary for us is an idea that has been around for years. Multimodal models have made it practical for the first time, paving the way for the first public release of the BYO-Me app, now available for iPhone and Android.

The task hides two different problems. The first is **recognising**: saying that the plate holds pasta with radicchio, courgettes and an apple. It is a classification problem, and modern models are getting better and better at it. The second is **quantifying**: estimating that the pasta weighs 262 grams. From a single two-dimensional photo, with no reference object for scale and no depth information, the height of a heap of rice or the water level in a clear cup are inherently ambiguous.

For a nutrition app the second problem is the one that matters: calories and nutrients depend on grams, not on the name of the dish.

That is why, before building the BYO-Me platform, an in-depth study was carried out, which led to a peer-reviewed paper published in *Nutrients*.

## How the study was run

The dataset was built for the purpose. Six participants photographed their meals with their own smartphones, mostly at the university canteen (166 images) and partly at home (34 images), with no prescribed shooting protocol, mimicking how future users and patients are expected to behave. Every visible item was **weighed on a kitchen scale**, net of the plate: 870 items in all, one to seven per photo, with a median weight of 129 grams.

The study evaluated 17 open-source models in 26 configurations of different sizes, all under 25 billion parameters and 15 GB, run locally on Fastal's infrastructure with the [Ollama](https://ollama.com/) framework. Five proprietary frontier models served as a reference: GPT-4o, GPT-4.1, GPT-5.2 and GPT-5.5 by OpenAI, and Claude Sonnet 5 by Anthropic.

Two metrics guide the reading of the results:

- **F1** measures recognition: it combines how many of the real items the model found and how many of the items it named are actually on the plate. It equals 1 when the model lists exactly the foods present, missing none and inventing none — the so-called *hallucinations*.
- **NMAE** measures the error on weight, in grams, and also penalises the items the model failed to recognise. It therefore goes beyond the mean error on the items found: a model that spots only a few items but weighs them well still does not get a good score.

Since a model may write "boiled spinach" where the label says "cooked spinach", matching answers to labels is done by a semantic comparison system, itself **validated against three human raters** on 300 sample pairs. All results come with confidence intervals computed by bootstrapping over images, and with independent repeated runs.

## The key findings

### Qwen2.5-VL 7B leads open-source recognition

In the benchmark published in *Nutrients*, Qwen2.5-VL 7B achieves the best recognition among open-source models, with F1 = 0.68 (95% confidence interval: 0.66–0.71), ahead of every other open model in virtually all resamples. Qwen3.5 9B and MiniCPM-V 4.5 follow, both at 0.65. Among proprietary models the best is GPT-4o, at 0.77.

An F1 of 0.68 does not mean that 68% of meals were recognised perfectly. The authors estimate that 70 to 81% of the accepted matches refer to the same food, while the rest are similar foods or, less often, different ones. Counting only exact matches, the F1 of Qwen2.5-VL 7B drops to 0.56 and that of GPT-4o to 0.59, but the ranking does not change. Absolute values should therefore be read as an upper bound; the comparison between models is solid.

![Horizontal bar chart of F1 with 95% confidence intervals. Open-source: Qwen2.5-VL 7B 0.68, Qwen3.5 9B 0.65, MiniCPM-V 4.5 0.65, Qwen3.5 4B 0.62, Llama 3.2-Vision 0.60, Mistral Small 3.1 0.55, Gemma 4 12B 0.53. Proprietary, with OpenAI models in release order: GPT-4o 0.77, GPT-4.1 0.75, GPT-5.2 0.67, GPT-5.5 0.66; Claude Sonnet 5 0.71.](/images/blog/ai-nutrizione-riconoscimento-en.svg "Figure 1 — Food recognition: the seven best open-source models and the five proprietary references. Among OpenAI models, F1 drops with each generation.")

### Portion estimation remains the hardest challenge

On weight, the best open-source model is Qwen3.5 9B, with an NMAE of 70 grams; GPT-4o gets down to 58. The most telling result, though, is another one. The researchers built a reference predictor that does not look at the photo: it already knows which food is on the plate and assigns it that food's typical weight in the rest of the dataset. Its mean error is **35 grams**, and **no model does better** on the items it recognises: GPT-4o ties at 35 grams, Qwen3.5 9B reaches 41.

In other words, on this dataset the models' estimates are at best comparable to those of someone who knows the canteen's usual servings. The models are not completely blind to size: for the best of them, the estimate grows, weakly, with the actual weight of the portion. But that visual information is not yet precise enough to add anything to what the name of the food already suggests. The errors also follow a recognisable pattern: every model underestimates the largest portions, and water, with a median weight of about 330 grams, is almost always estimated at 200 or 250 — the capacity of a standard glass.

![Dot plot of eleven models. For each, the mean error of the model's estimate and of the typical serving on the same items: GPT-4o 35 vs 33 grams, Claude Sonnet 5 39 vs 33, Qwen3.5 9B 41 vs 35, GPT-5.5 42 vs 33, GPT-4.1 44 vs 34, Qwen3.5 4B 44 vs 34, GPT-5.2 45 vs 34, MiniCPM-V 4.5 47 vs 35, Mistral Small 3.1 48 vs 39, Gemma 4 12B 51 vs 33, Qwen2.5-VL 7B 54 vs 37.](/images/blog/ai-nutrizione-porzioni-en.svg "Figure 2 — Weight error compared, on the same items, with the typical serving: in no row does the model do better.")

### Newer does not automatically mean better

The least intuitive result concerns model generations. Among OpenAI's proprietary models, recognition runs in reverse release order: GPT-4o 0.77, GPT-4.1 0.75, GPT-5.2 0.67, GPT-5.5 0.66, with gaps far larger than the run-to-run variability. The authors trace the main cause to newer models splitting a dish into more entries than a nutritionist annotated — for instance "rigatoni", "black olives" and "diced tomatoes" instead of "pasta with tuna, olives and tomatoes". More unmatched entries mean lower precision.

On portions, by contrast, newer generations improve: from Qwen2.5-VL 7B to Qwen3.5 9B the mean error on recognised items falls from 54 to 41 grams, about 25% less. And among open-source models the recognition ranking depends on how it is scored: with a second judge, an LLM that uses no similarity threshold, the successors match or overtake Qwen2.5-VL 7B. The authors' practical conclusion is that the newest model is not necessarily the best choice, and that the choice should be validated on the specific task.

## Fastal's contribution and the collaboration

Fastal took part in the project as a technology partner. It provided the **IT infrastructure** that hosted the open-source models and the experimental tooling, and **consultancy on LLM technologies** and the related software frameworks. That is the work that makes a benchmark of this size possible: dozens of configurations, six sampling temperatures and repeated runs, each on the same 200 images and with recorded model versions, so that the results can be reproduced.

In the interest of transparency: the author of this article, Stefano Capezzone, CEO and co-founder of Fastal, is one of the authors of the paper. The study design, the analysis and the interpretation of the results are the responsibility of the authors.

The research is led by the Metabolic Intelligence Lab at Università Cattolica del Sacro Cuore, directed by Prof. Giuseppe Maulucci, and funded by a university grant. It shows how academic expertise in nutrition and scientific validation can work on the same problem as industrial expertise in deploying models.

## What it means for digital health and for AI builders

For anyone designing digital nutrition apps, the study shows where general-purpose models are already useful and where they are not. Automatically listing what is on the plate is a realistic goal; deriving the grams from a single photo, with the precision a clinical use requires, is not yet. The authors point to fine-tuning open-source models on food data, using scale references or depth information, and integrating 3D estimation models.

The lesson applies beyond nutrition, and it is the same one that emerges from our work on [Document Intelligence with Vision LLMs and OCR](/en/blog/2026-09-vision-llm-ocr-kyc-aml/). What separates a convincing prototype from a reliable system is evaluation: real data relevant to the use case, metrics that do not hide errors, and human validation of the very way things are measured. A generic leaderboard does not tell you which model will work on your problem.

There is also an infrastructure angle. All the open-source models in the benchmark are under 15 GB and run locally. The study does not measure cost or latency, but it is a reminder that models of this size can run on-premise or on the device, without sending photos of what someone eats to external services — a topic we also explored by [running an LLM on a Raspberry Pi](/en/blog/2025-10-linux-day-llm/).

## Frequently asked questions

### Can a Vision LLM recognise the food in a photograph?

Yes, with performance that varies widely from model to model. In the benchmark published in *Nutrients* in 2026, on 200 photos of real meals, the best open-source model, Qwen2.5-VL 7B, reaches F1 = 0.68 and the best proprietary one, GPT-4o, 0.77. The values depend on the criterion used to decide that the model's answer matches the real food: counting only exact matches, they drop to 0.56 and 0.59 respectively.

### Can it accurately measure the weight of a portion from a single photo?

Not reliably. In the study no model estimates the weight of the items it recognises better than a predictor that already knows the food and assigns it its typical serving, with a mean error of 35 grams. Every model underestimates the largest portions. The authors stress that an estimate derived from a photo is not a physical measurement.

### Which open-source model performed best?

It depends on the task. For recognition the best is Qwen2.5-VL 7B (F1 = 0.68); for portion estimation it is Qwen3.5 9B (NMAE = 70 grams), which however lists more non-existent items and loses the lead when those are penalised too. None of the 26 open-source configurations evaluated matches GPT-4o on both tasks.

### What was Fastal's role in the research?

Fastal provided the IT infrastructure that hosted the open-source models and the experimental tooling, and consultancy on LLM technologies and the related frameworks. The study design, the analysis and the interpretation of the results belong to the authors, who include Fastal's CEO, Stefano Capezzone.

## Open research and next steps

The dataset and code are public in the [Metabolic Intelligence Lab repository](https://github.com/Metabolic-Intelligence-Lab/vlm-food-benchmark): the 200 photographs, the 870 annotations, the raw responses of every model and the full analysis pipeline, under the MIT licence for the code and CC BY 4.0 for the data. Anyone can redo the numbers or test a new model on the same benchmark.

The authors state the limitations themselves. The dataset comes from a single canteen and a single cuisine, Italian, with standardised servings that make the "typical serving" an unusually strong reference; on home-cooked meals every model performs worse. The study is not a clinical validation and does not measure calories or nutrients. The next steps are a larger dataset covering more cuisines and settings, and specialising open-source models for the food domain.

**Scientific source.** Marchetti, T.; Capezzone, S.; Esposito, R.; De Spirito, M.; Maulucci, G. *Benchmarking Open-Source Vision–Language Models for Food Image Recognition and Portion Size Estimation.* Nutrients 2026, 18(21), 3582. [https://doi.org/10.3390/nu18213582](https://doi.org/10.3390/nu18213582)

*This article is based on a peer-reviewed scientific publication: results and interpretations refer to that specific benchmark.*

To dig deeper, [read the full study in Nutrients](https://doi.org/10.3390/nu18213582) or find out how we approach [artificial intelligence](/en/services/artificial-intelligence/) projects.