Manuscript in preparation

Memory-Augmented Crop-Disease Evaluation of Frozen Vision-Language Models

Tirtho Roy1,*, Shreyan Ganguly1,*, Muhammad Arbab Arshad1, Nikhil Bhattacharya1, Soumik Sarkar1,✉
1Iowa State University

* Equal contribution   ✉ Corresponding author: soumiks@iastate.edu

10
Crops
1,600
Test queries
+19.4
Points over SAGE, in distribution
+6.3
Points over SAGE, held-out collections
Mango disease image
Mango
Cucumber disease image
Cucumber
Corn disease image
Corn
Wheat disease image
Wheat
Potato disease image
Potato
Rice disease image
Rice
Grape disease image
Grape
Apple disease image
Apple
Tomato disease image
Tomato
Soybean disease image
Soybean
Central Hypothesis
Caption memory adds diagnostic information beyond label lookup. A frozen vision-language model (VLM) with a few retrieved, labeled captions should therefore exceed both our earlier SAGE system and a nearest-neighbor label lookup, in distribution and on image collections it has never seen.
TL;DR The hypothesis holds in part. MACE reaches 62.2% in distribution and 57.0% on held-out collections, above SAGE (42.8% and 50.7%) and DINOv2 label lookup (52.5% and 38.2%). Retrieved labels account for about two thirds of the gain, caption text adds 6.4 points in distribution in a post hoc test, and accuracy falls below SAGE when the true disease is not retrieved. Claude Haiku 5.5, reading the same Sonnet captions, reaches 55.5% and 53.5%. It beats SAGE (37.0% and 42.5%) but does not clearly beat DINOv2 lookup in distribution. The strongest gain is in distribution, and the held-out gain depends on the source: 0.0 with Sonnet and +6.4 with Haiku on the seven crops that hold out Bugwood, against +21.1 and +21.7 on the other three. A confidence fallback did not help.
Overview

Abstract

Crop disease diagnosis is critical for food security, yet labeled disease images are limited and models trained on curated images generalize poorly to field images. We showed in AgEval that general-purpose vision-language models (VLMs) adapt to plant stress tasks from a few annotated examples, and we gave frozen VLMs source-grounded symptom knowledge for 335 crops in SAGE. Building on both studies, we introduce MACE (Memory-Augmented Crop-Disease Evaluation), a training-free method that adds a caption memory to SAGE. Each labeled reference image is described once as text, and the four most relevant labeled captions are retrieved for each test image. We evaluate MACE against label lookup on ten crops and 1,600 queries with two frozen diagnosis VLMs that share Sonnet-written captions, Claude Sonnet 5.5 and Claude Haiku 5.5, with near-copies removed and one image collection held out per crop. With four memory images per disease, MACE reaches 62.2% accuracy in distribution and 57.0% on held-out collections with Sonnet, and 55.5% and 53.5% with Haiku. SAGE reaches 42.8% and 50.7% with Sonnet (MACE gain +19.4 and +6.3 points) and 37.0% and 42.5% with Haiku (+18.5 and +11.0). Nearest-neighbor label lookup with DINOv2 reaches 52.5% and 38.2%, so Haiku with MACE does not clearly exceed it in distribution. With both models, retrieved labels account for about two thirds of the gain and caption text adds 6.4 points in distribution in post hoc tests. The strongest gain is in distribution, and the held-out gain depends on the source: it is 0.0 with Sonnet and +6.4 with Haiku on the seven crops that hold out Bugwood, and +21.1 and +21.7 on the three crops that hold out other collections. The held-out evidence is therefore source-specific. With both models, MACE falls below SAGE when the true disease is not retrieved, and a retrieval-confidence fallback did not improve accuracy. With Sonnet, MACE adds $0.0014 per query over SAGE and, at about one fifth of the price, reaches higher accuracy than an agent reading the same captions.

Method

Building on AgEval and SAGE

What we build on

In AgEval, we showed that general-purpose VLMs adapt to plant stress tasks from a few annotated examples. In SAGE, we gave frozen VLMs source-grounded symptom knowledge for 335 crops.

What MACE adds

MACE describes each labeled reference image once as a label-free caption. For each test image, the four most similar captions are retrieved with their labels and added to the SAGE knowledge. The VLM is not trained.

MACE overview on a recorded tomato query

MACE on a recorded query, from offline caption memory to online retrieval and diagnosis.

Results

Five questions, five answers

Q1Across cropsYes, on crop means

Does MACE improve diagnosis across crops and under collection shift?

42.8→62.2SAGE to MACE accuracy (%), in distribution

The held-out gain is smaller and varies by crop. On held-out collections, MACE is higher than SAGE on five crops, equal on one and lower on four. In a post hoc grouping, the gain is 0.0 points on the seven crops that hold out Bugwood and +21.1 on the other three, and without mango and rice it is +0.6. The same seven crops gain +18.7 in distribution, so the drop follows the source more than the crop. Haiku gains +18.5 and +11.0 overall, and +6.4 and +21.7 on the two held-out groups. Sonnet MACE is higher than DINOv2 lookup on all ten held-out collections.

+6.3 points over SAGE, held-out+18.8 points over DINOv2, held-out
Q2EvidenceMostly labels

Which evidence produces the gain?

63.0
Captions and labels
60.0
Labels only
46.5
Captions only

Accuracy (%) on 400 ablation queries, in distribution. On those queries, labels only did not differ from MACE. A post hoc test on all queries shows that caption text adds 6.4 points in distribution; the held-out difference is inconclusive. Captions add +6.3 points when DINOv2 selects the memory.

Q3Single answersMore fixed than broken

How does memory change single answers?

22.6%vs3.2%SAGE errors corrected against new errors, in distribution

When the true disease is not retrieved, MACE falls below SAGE: 16.0% against 23.1% in distribution, and 24.4% against 33.1% on held-out collections.

Q4RobustnessDepends on memory

How robust is the gain?

1.7Points of SD across three memory selections, in distribution

Accuracy rises with memory size, from 53.5% to 65.5% in distribution between one and sixteen images per disease. Memory from the same collection still helps, by +5.5 points at equal bank size and disease mix; the cause is unknown.

Q5PriceSmall increase

What does the gain cost?

57.5% at $0.017
MACE retrieval
48.5% at $0.020
Image references
48.0% at $0.091
MACE agentic

On 200 matched queries, MACE adds 22.5 points over SAGE for $0.0014 more per query. Prices are list-price estimates in US dollars.

Figures

From crops to single answers

MACE gain over SAGE per crop for Sonnet and Haiku

MACE gain over SAGE per crop. Both VLMs, with paired 95% intervals. The dashed line separates the seven crops that hold out Bugwood from the other sources.

Evidence controls

Which evidence matters? Planned controls, 400 queries.

Per-crop effects of caption text and caption retrieval

Where captions help, per crop, in points (post hoc).

Two recorded retrieval traces

Recorded traces, one success and one failure, illustrative. Steps are fixed stages, not agent actions.

Answer outcomes and accuracy by retrieved evidence

Answer outcomes and accuracy by retrieved evidence.

Robustness panels

Robustness to memory size, selection and source.

Accuracy, price and latency

Accuracy and cost on 200 matched queries.

Numbers

Main results

MethodIn distributionHeld-outMacro, in distributionMacro, held-out
No VLM call
Label majority vote52.6 [49.6, 55.7]41.5 [37.8, 45.0]39.132.3
Score-weighted vote55.7 [52.7, 58.7]44.5 [40.8, 48.2]45.035.8
OpenCLIP 1-NN54.7 [51.7, 57.7]37.7 [34.0, 41.3]45.532.4
DINOv2 1-NN52.5 [49.6, 55.5]38.2 [34.5, 42.0]41.632.0
Frozen VLM: Sonnet 5.5
SAGE (KB only)42.8 [39.8, 46.0]50.7 [46.8, 54.3]34.040.0
MACE retrieval62.2 [59.3, 65.0]57.0 [53.2, 60.7]51.845.6
Post hoc tests, frozen VLM
Labels only55.8 [52.9, 58.8]54.8 [51.0, 58.7]47.342.5
DINOv2-retrieved labels55.2 [52.2, 58.2]55.8 [51.8, 59.7]45.244.5

Equal-crop accuracy (%) with 95% bootstrap intervals, four memory images per disease. Macro: mean per-disease recall. Post hoc tests were added after the labels-only control was read.

Follow-up tests

Stress tests of the main result

TestIn distributionHeld-out
Second diagnosis model: Claude Haiku 5.5, same Sonnet captions
SAGE37.0 [34.0, 39.9]42.5 [38.8, 46.2]
Labels only49.1 [46.2, 52.3]50.3 [46.5, 54.0]
MACE55.5 [52.5, 58.5]53.5 [49.8, 57.2]
Retrieval and evidence, Sonnet 5.5
DINOv2 retrieval with captions61.5 [58.6, 64.3]56.3 [52.5, 60.0]
Fallback to SAGE at low retrieval confidence, Sonnet 5.5
Frozen fallback rule61.8 [58.9, 64.6]57.8 [54.2, 61.5]
Labeled oracle (upper bound only)63.4 [60.5, 66.2]59.3 [55.5, 63.2]
Memory source, 554 in-distribution queries with the true disease in memory, Sonnet 5.5
Standard memory60.7 [56.5, 64.9]not run
Same size and disease mix61.2 [56.9, 65.3]not run
Other collections only55.7 [51.6, 59.6]not run

Equal-crop accuracy (%) with 95% bootstrap intervals. All tests were added after the main results and are not preregistered.

Second model

Haiku MACE exceeds SAGE by +18.5 points in distribution and +11.0 held-out. It does not clearly exceed DINOv2 lookup in distribution (+3.0 [−0.8, +6.6]), although it does held-out (+15.3 [+10.5, +20.3]). Haiku tests a second diagnosis model, not an independent pipeline: it reads Sonnet captions, and both are Claude models.

Captions as evidence

Captions add +6.3 points in distribution when DINOv2 selects the memory, close to the +6.4 under BM25. Held-out caption effects are inconclusive under both retrievers.

Fallback did not help

A rule frozen on 600 separate images changed accuracy by −0.4 [−1.0, +0.1] in distribution and +0.8 [−0.5, +2.2] held-out. Even an oracle would add only 1.2 and 2.3 points.

Memory source matters

Removing same-collection images costs 12.8 points, mostly because 45% of queries lose their disease from memory. With the disease kept, a bank of equal size recovers the 5.0-point loss (+5.5 [+1.9, +9.1]). The cause is unknown: it may be a shortcut such as shared background, or closer disease appearance.

Uneven held-out gain (post hoc)

Sonnet gains 0.0 [−3.8, +3.8] on the seven crops that hold out Bugwood and +21.1 [+13.9, +27.8] on the other three. Haiku gains +6.4 [+1.7, +11.0] and +21.7 [+15.0, +28.9]. With crops as the unit, the held-out interval is [−3.2, +15.9] for Sonnet and [+4.2, +17.8] for Haiku. Seven crops share one source, so this is source-specific evidence.

Accuracy by Claude model, and per-crop gains

Two Claude models, with per-crop gains.

Retrieval by evidence accuracy, and caption effect by crop

Retrieval by evidence, with caption effects by crop.

Fallback calibration and final accuracy

Fallback threshold sensitivity and final accuracy.

Accuracy by memory source, and held-out gain against image shift

Memory source by crop, and held-out gain against image shift.

Data, statistics and the full report are in results/extension.

Discussion

What this means

Retrieval failure

MACE gains 24.8 and 11.8 points over SAGE on retrieval hits and loses 7.1 and 8.8 points on misses. A label-free fallback to SAGE at low retrieval confidence fired on 4.1% and 19.2% of queries and did not detectably change accuracy with either model, and even an oracle would add at most 2.3 points. It was calibrated on 600 images outside the test queries, of which 93 sit in memories that serve test queries.

Roles of a caption

Caption text added accuracy in distribution. Captions added as much after DINOv2 retrieval as after BM25, so in distribution the caption acts mainly as evidence and not as an index.

Limitations

  • The caption, DINOv2, Haiku, fallback and collection tests are post hoc.
  • Each result is one run, and both VLMs are Claude models sharing Sonnet captions. Haiku ran only SAGE, MACE and labels only.
  • Seven crops share one held-out collection, so crop and collection shift remain confounded.
  • Ten crops do not represent all of agriculture.
  • No plant pathologist reviewed the labels or recorded cases.
People

Team

Tirtho Roy
Iowa State University
Equal contribution
Shreyan Ganguly
Iowa State University
Equal contribution
Muhammad Arbab Arshad
Iowa State University
Nikhil Bhattacharya
Nikhil Bhattacharya
Iowa State University
Soumik Sarkar
Iowa State University
Corresponding author

BibTeX

@unpublished{roy2026mace,
  title  = {Memory-Augmented Crop-Disease Evaluation of Frozen Vision-Language Models},
  author = {Roy, Tirtho and Ganguly, Shreyan and Arshad, Muhammad Arbab and Bhattacharya, Nikhil and
            Sarkar, Soumik},
  note   = {Manuscript in preparation},
  year   = {2026}
}
Rights

License and copyright

This page

© 2026 the MACE authors. Original text, charts, diagrams and the MACE logo are licensed under CC BY-SA 4.0. Full terms are in LICENSE.md.

Crop images

Each photograph stays under the license of its source and was cropped and resized for display. The mango image may not be used commercially. Team photographs are not licensed for reuse.

Rights concerns

To report a credit or rights concern, contact soumiks@iastate.edu. Product and model names belong to their owners.

ImageImage idSourceLicense
Crop strip, Soybeans14_r3391Soybean stress images, Ghosal et al., PNAS 2018CC BY 4.0
Crop strip, Corns4_r8651PlantDoc, Singh et al., CoDS-COMAD 2020CC BY 4.0
Crop strip, Tomatos12_r18265LeafNet, Quoc et al., 2026CC BY 4.0
Crop strip, Cucumbers16_r4806Cucumber Disease and Freshness, Menco Tovar et al., Zenodo 2025CC BY 4.0
Crop strip, Wheats15_r6038VQA Plant-Disease Merged, Raghavendrad60, Hugging FaceApache 2.0
Crop strip, Mangos39_r21419MangoLeafBD, Ahmed et al., Data in Brief 2023CC BY-NC 3.0
Crop strip, Apples20_r10262PlantDoc, Singh et al., CoDS-COMAD 2020CC BY 4.0
Crop strip, Grapes1_r4819PlantVillage, Hughes and Salathé, 2015CC BY-SA 3.0
Crop strip, Rices29_r6668VQA Plant-Disease Merged, Raghavendrad60, Hugging FaceApache 2.0
Crop strip, Potatos47_r140PlantDoc, Singh et al., CoDS-COMAD 2020CC BY 4.0
Success trace, test images15_r12385Cucumber Disease and Freshness, Menco Tovar et al., Zenodo 2025CC BY 4.0
Success trace, reference 1s16_r4423LeafNet, Quoc et al., 2026CC BY 4.0
Success trace, reference 2s15_r12256Cucumber Disease and Freshness, Menco Tovar et al., Zenodo 2025CC BY 4.0
Success trace, reference 3s15_r12371LeafNet, Quoc et al., 2026CC BY 4.0
Success trace, reference 4s15_r12553Cucumber Disease and Freshness, Menco Tovar et al., Zenodo 2025CC BY 4.0
Failure trace, test images16_r491Cucumber Disease and Freshness, Menco Tovar et al., Zenodo 2025CC BY 4.0
Failure trace, reference 1s15_r12604LeafNet, Quoc et al., 2026CC BY 4.0
Failure trace, reference 2s15_r12665Cucumber Disease and Freshness, Menco Tovar et al., Zenodo 2025CC BY 4.0
Failure trace, reference 3s16_r4446LeafNet, Quoc et al., 2026CC BY 4.0
Failure trace, reference 4Image withheld: its source does not permit adapted copies. Only the label is shown.

License status as recorded on each source page, for the images shown on this page.