Heuristic Learning for Image Classification: A Symbolic Phase 1 Without Neural Networks

A coding agent maintains a symbolic visual system that keeps improving: no gradients, no backpropagation, no forgetting.

The Provocation

Jiayi Weng's Learning Beyond Gradients [1] makes a striking claim: heuristics were never useless, they were just too expensive to maintain. Coding agents change that maintenance curve. A rule-based system maintained by an LLM agent can keep absorbing feedback, growing more capable, and never catastrophically forgetting, because old capabilities live in code and tests, not in weight matrices that get overwritten.

Weng demonstrated this with Atari (Breakout reaching the theoretical maximum 864), MuJoCo (Ant at 6000+), and an Atari57 batch where median human-normalized scores rivaled PPO baselines. The core insight: what's being updated is not a policy function but a software system, with memory, feedback channels, regression tests, and experiment records.

But Weng also wrote: "With what I know today, I cannot imagine an agent writing pure Python code, without a neural network, to solve ImageNet."

I decided to test that boundary directly.

This Experiment: Heuristic Learning Applied to Vision

Task: Classify 64x64 ImageNet images into 10 classes: golden retriever, mushroom, teapot, school bus, banana, bicycle, eagle, laptop, piano, zebra.

Method: A coding agent (Claude) iteratively builds and maintains a purely symbolic image classification system. No neural networks. No embeddings. Only classical computer vision (OpenCV), hand-crafted features, and symbolic scoring rules. (The system does store two histogram prototypes computed from training data for one tiebreaker, and all thresholds were tuned against the eval set, so "no learned parameters" would be misleading. But there is no gradient descent, no backpropagation, and no weight matrix anywhere.)

Process: The exact HL loop Weng describes:

run evaluation → analyze confusion → propose feature/fix → test for regressions → deploy → repeat

Over 11 sessions and 248 evaluation runs, the system grew from a skeleton into a 2500-line Heuristic System containing 40 registered features, 22 pairwise tiebreaker functions, a hierarchical class taxonomy, regression-tested scoring rules, and complete experiment logs.

Result: 86.1% top-1 accuracy on the 230-image development set (see methodology note below). The original design target was >50%.

Methodology Note: How the 86.1% Was Computed

Full transparency on the evaluation:

The 86.1% (198/230) was measured on the same 230 images used during development. All thresholds were tuned against this set. This is a development-set accuracy, not a held-out validation accuracy.

The 230-image set breaks down as:

• 4 "hard" classes (golden retriever, mushroom, teapot, school bus): 50 real Tiny ImageNet images each = 200 images. These scored 84% (168/200) collectively.
• 6 "easy" classes (banana, bicycle, eagle, laptop, piano, zebra): 5 synthetic (AI-generated) images each = 30 images. These scored 100% (30/30) — but these are trivial: the banana images are identical yellow ellipses on grey noise, and the other synthetics are similarly artificial.

The 6 easy classes inflate the headline number from 84% to 86.1%. On a strict reading, this is a 4-class classifier at 84% on its own tuning set.

However, the synthetic classes were not pure decoration — they served as negative-class scaffolding. The system's excluding features (e.g., striped_texture penalizing teapot, keyboard_pattern penalizing golden retriever, wheel_like penalizing mushroom) were developed precisely because these classes existed as scoring alternatives. The decision boundary shaping that comes from 10-class competition is architecturally meaningful even when 6 of those classes have trivial evaluation images.

When we ran the classifier (unchanged) on a 400-image validation folder for the 4 real classes, the accuracy dropped to 54%. Exact file-hash checking later found that 38 of those 400 images overlapped with the development set. On the stricter non-overlapping subset, accuracy was 51.4% (186/362). Since this validation subset has 4 classes, the clean random baseline is 25%, so the validation result is about 2.1x random. The gap reflects threshold overfitting to the 50 development images per class.

What Phase 1 demonstrated regardless: The HL loop itself works. The trajectory from 12.7% to 84% on hard classes shows genuine iterative improvement through confusion-driven feature invention. The analysis of plateau-breaking moments, coupling complexity, representation saturation, and the single-swap constraint remains valid. Phase 1 is a proof-of-concept for the methodology; Phase 2 is the proper evaluation.

We are currently running a Phase 2 experiment with a proper train/val split to get honest generalization numbers. See the repository README for updates.

Why This Is Heuristic Learning

Let me map this experiment directly onto Weng's HL framework:

The object being maintained is not a model. It's a Heuristic System in Weng's precise sense: a programmatic policy, state representation, feedback channels, experiment records, and an update mechanism executed by a coding agent. A single rule is not enough. Rules, feedback, history, and the next update path all connect.

The Update Loop

Each session followed this pattern:

1. Run evaluation (230 images, ~25ms each)
2. Analyze confusion (which classes are confused? what are the score gaps?)
3. Diagnose specific errors (read proof traces, measure pixel features on failing images)
4. Propose a fix (new feature, tiebreaker condition, scoring adjustment)
5. Test for regressions (does the fix break anything that was working?)
6. Deploy or discard (net positive → commit; net negative → revert and try something else)

This is exactly the HL feedback loop: environment feedback → coding agent reads context → edits policy → reruns → writes results back → continues to the next round.

The System Architecture

The Visual Concept Algebra

image (64x64 BGR)
  → 5 classical vision sensors (edges, color, texture, segmentation, shape)
  → ~30 symbolic atoms per image
  → 40 registered features across 5 categories
  → hierarchical flat scorer
  → pairwise tiebreaker system
  → prediction with full proof trace

Every prediction produces a human-readable explanation:

Claim: image contains 'golden_retriever'
Route: root → animal → golden_retriever
Evidence:
  golden_brown_color: 0.95 (hue 8-35, sat>40 in 67% of pixels)
  organic_texture: 0.78 (14/25 patches with entropy>2.0)
  green_context: 0.62 (35% green pixels in background)
Absent (supporting exclusion):
  striped_texture: not detected
  keyboard_pattern: not detected

The Scoring Formula

Each class has three feature lists:

• Required (AND-gate): ALL must fire or the class scores zero
• Supporting: averaged together, weighted 0.3
• Excluding: averaged together, penalizes 0.2

score = required_avg × 0.6 + supporting_avg × 0.3 − excluding_avg × 0.2

A golden retriever requires golden_brown_color. A zebra requires both striped_texture AND pure_vertical_stripes. If any required feature doesn't fire, the class is eliminated.

The Growth Trajectory

Session 1:   ~20%   baseline sensors + features
Session 2:    35%   hierarchy + scorer
Session 3:    44%   compound features + tiebreakers
Session 4:    57%   tiebreaker expansion
Session 5:    62%   spatial attention + synthetic tiebreakers
Session 6:    67%   eagle/banana solved
Session 7:    68%   plateau confirmed (DCT explored, failed)
Session 8:    78%   banana cap + compound conditions
Session 9:    80%   continued tiebreaker refinement
Session 10:   85%   alt required features + guard tightening
Session 11:   86%   green+warm counter-signals (final)

This curve mirrors Weng's Breakout trajectory (387 → 507 → 839 → 864): rapid early gains from structural changes, then increasingly targeted fixes at the margin. The Breakout policy grew "action probes, state readers, ball and paddle detectors, landing prediction, stuck-loop detection, regression tests, video replays, and experiment logs." My classifier grew feature detectors, tiebreaker functions, confusion diagnostics, score caps, margin thresholds, and conjunctive guard conditions.

Both systems grew into something much more than a single policy file. Both required the coding agent to maintain a system, not just write one rule and move on.

The HL Properties in Practice

Weng identifies several properties that HL has over Deep RL. Here's how each manifested:

Explainability: How the System Actually Decides

Every classification produces a full proof trace: which features fired, what evidence supported them, what was absent. No saliency map, no gradient attribution, no post-hoc explanation. The reasoning is the decision.

Here are real decision traces from the final system:

A zebra (correct, high confidence):

Prediction: zebra (0.701)
Alternatives: piano (0.66), teapot (0.42)

Claim: image contains 'zebra'
Route: root -> animal -> zebra
Evidence:
  pure_vertical_stripes: 1.00 (col_var=122.2, row_var=0.3)
  black_white_dominant:  1.00 (BW coverage: 0.98; low saturation: True)
  striped_texture:       0.90 (consistency=1.00, energy=0.67, bw=0.98)
Absent (supporting exclusion):
  green_context: not detected
  wheel_like: not detected

The system says: "I see strong vertical stripes (column variance 122 vs row variance 0.3), the image is 98% black and white, and stripe consistency is 1.0. No green background (not outdoor animal), no wheels (not bicycle). This is a zebra." A human can verify each claim by looking at the image.

A golden retriever (correct, tight margin):

Prediction: golden_retriever (0.751)
Alternatives: teapot (0.43), mushroom (0.40)

Evidence:
  golden_brown_color:    1.00 (golden/brown coverage: 0.58)
  golden_fur_in_nature:  1.00 (golden=0.58, green=0.06, var=2966)
  large_warm_blob:       1.00 (dominance=1.00, coverage=0.71)
  outdoor_animal_scene:  1.00 (nature=0.71, var=1134)
Absent:
  striped_texture: not detected
  repeated_vertical_lines: not detected

The system says: "58% of pixels match golden-brown hue (hue 8-35, sat>40). There's a large warm-colored blob covering 71% of the image. The scene has nature context (0.71). No stripes, no vertical lines. This is a golden retriever." Note that teapot scored 0.43, only 0.32 behind. The margin reflects real ambiguity: at 64x64, a brass teapot can look similar.

A school bus (correct, won via tiebreaker):

Prediction: school_bus (0.502)
Alternatives: teapot (0.46), mushroom (0.38)

Evidence:
  yellow_body_with_sky:  1.00 (sky=0.95, yellow_body=0.13)
  sky_above_object:      0.95 (sky detected in top: ratio=0.95)
  repeated_vertical_lines: 0.82 (21 crossings, amplitude=35)
  horizontal_window_pattern: fires (dark rectangular bands)

The system says: "95% sky in the top quarter, yellow body visible below. 21 vertical luminance crossings in the middle band (window pattern). This is a school bus." The horizontal_window_pattern feature is pathognomonic: no other class has a regular grid of dark rectangles.

A mushroom (correct, won via tiebreaker swap):

Prediction: mushroom (0.446)
Runner-up before swap: golden_retriever (0.724)

Evidence:
  golden_brown_color:  1.00 (coverage: 0.69)
  large_warm_blob:     1.00 (coverage: 0.69)
Tiebreaker: dog_vs_mushroom
  grad_top_bin = 0.21 (> 0.18 threshold)
  green_ratio = 0.38 (> 0.089 threshold)
  warm_ratio = 0.69 (< 0.68 threshold? no, but green > 0.25)
  -> directional gradients + green context -> swap to mushroom

This is the hard case. The mushroom's brown cap fires golden_brown_color at 0.69, making golden_retriever score 0.724 as the base winner. But the tiebreaker detects: (1) directional gradient concentration (cap surface has oriented texture), (2) 38% green pixels (forest floor), (3) warm coverage 0.69 but with green context. The conjunction of all three signals triggers a swap.

A failure: teapot_0027 misclassified as golden_retriever:

Prediction: golden_retriever (0.737)   [WRONG - true label: teapot]
Runner-up: teapot (0.458)

Evidence:
  golden_brown_color:   1.00 (golden/brown coverage: 0.34)
  golden_fur_in_nature: 0.97 (golden=0.31, green=0.01, var=7670)
  large_warm_blob:      1.00 (dominance=0.91, coverage=0.39)
  outdoor_animal_scene:  1.00 (nature=0.46, var=9156)
  organic_texture:      0.79 (15 patches)
Absent:
  striped_texture: not detected
  green_context: not detected

The failure is readable. The brass teapot's warm surface matches golden-brown hue at 34% coverage, and the warm blob covers 39% of the image with 0.91 dominance. The tiebreaker (dog_vs_teapot) checks for green outdoor context (0.01, far below the 0.30 threshold for a dog counter-signal) and background contrast (not strong enough to flip). The system correctly identifies why it fails: brass and fur share the same HSV range at 64x64, and this particular teapot lacks the distinct-object-on-background signature that would override.

This level of transparency is impossible with neural networks. A ResNet achieving 95% on this task would give you a softmax vector. If it misclassifies a brass teapot as a dog, you get "dog: 0.73, teapot: 0.21" with no explanation of what visual evidence led to that decision.

Sample Efficiency

When the coding agent identified that green_ratio > 0.50 AND warm_ratio > 0.34 AND yellow_ratio < 0.45 separates dogs-in-nature from mushrooms, that single code edit jumped the policy directly to a new level: three previously failing images fixed in one commit. A neural network would need many gradient steps and risk catastrophic forgetting.

Regression-Testability

Every fix was regression-tested against all 230 images. Old capabilities become implicit test cases. When I added a dog-vs-mushroom condition, I immediately checked whether any correctly-classified mushrooms flipped to dog. This is the HL version of "old capabilities can become tests, replays, or golden cases."

In one instance, adding warm_ratio > 0.20 as a dog-vs-mushroom signal caused mush_0001 to regress (it had warm=0.248, just above threshold). The fix was tightening to warm_ratio > 0.34. This kind of precise threshold surgery is possible because the policy is code, not a 40M-parameter weight matrix.

Avoiding Catastrophic Forgetting

The system never forgot how to classify zebras when I improved mushroom detection. Features and tiebreakers are modular: changing one class's scoring doesn't affect another's unless they share features. Old capabilities literally live in their own code paths.

This is Weng's key insight: "old capabilities do not have to live only inside model weights; they can be written into rule sets and tests."

What Worked and What Failed

Worked: Pairwise Tiebreakers (The Core HL Mechanism)

After the base scorer ranks all 10 classes, the top-4 candidates are checked pairwise. If two candidates are within a margin threshold AND a specialized pixel-level function determines the lower-ranked one should win, they swap.

This is responsible for ~15% of the final accuracy. It's the HL equivalent of Weng's Breakout "stuck-loop detection": a targeted mechanism that handles a specific failure mode.

The critical constraint: only one swap per prediction. Multi-swap causes cascading pathological chains (-2% regression). This mirrors Weng's observation that HL systems need bounded local corrections, not unbounded global rewrites.

Worked: Conjunctive Conditions

No single feature separates golden retrievers from mushrooms at 64x64. But conjunctions of 3-4 weak signals work:

• green > 0.50 AND warm > 0.34 AND yellow < 0.45 → dog
• gradient_top_bin > 0.18 AND green > 0.089 AND warm < 0.68 → mushroom
• texture_ratio > 2.5 AND bright_diff < 0 → teapot

This is the HL equivalent of Weng's Ant policy growing "rhythmic control, posture feedback, contact signals, and short-horizon model rollouts" , individually simple rules that compose into effective behavior.

Failed: Every Global Optimization

Score normalization (-3%), z-scores (regression), calibration (regression), stronger excluding penalty (-2.6%). Every attempt to improve the system globally destroyed carefully tuned local interactions.

This validates Weng's framework: HL updates work locally. The coding agent makes targeted edits. Sweeping parameter changes are the symbolic equivalent of learning rate disasters in Deep RL.

Failed: Alternative Swap Mechanisms

"Demote" instead of swap: 72% (-14%). Post-swap re-sort: 55% (-31%). These are architecturally "cleaner" but they destroy the emergent properties of the existing system.

This is the HL coupling-complexity problem. The system's modules interact. A "better" abstraction that ignores these interactions fails catastrophically. Weng's concept of coupling complexity ("how many interdependent states, rules, tests, feedback signals, and historical constraints an update has to account for at the same time") is exactly what made these refactors fail.

The Representation Saturation Ceiling

The remaining 32 errors (out of 230) come from the dog/mushroom/teapot triangle: at 64x64, all three are "warm-colored smooth blobs." I measured every conceivable pixel-level feature:

This is representation saturation: all measurable properties of the signal have been captured, and no additional features can improve discrimination. The discriminative information (fur micro-texture, gill patterns, ceramic sheen) exists below the 64x64 Nyquist frequency.

In HL terms: the feedback loop has converged. Not because the coding agent can't write more code, but because the environment (the pixels) has stopped providing discriminative signal. The system needs a new representation (higher resolution) before more learning can happen.

This connects to Weng's Montezuma example: "Some environments need stronger program forms: composable macro-actions, recoverable search state, and long-term memory. Plain if else cannot solve everything." Here, the system needs a stronger sensory form (more pixels) before it can solve the remaining cases.

Addressing Weng's Challenge

Weng wrote: "I cannot imagine an agent writing pure Python code, without a neural network, to solve ImageNet."

At full ImageNet scale (1000 classes, 224x224), this is likely true. But the boundary is more nuanced than "impossible":

• 4 classes, 64x64, dev set: 84% (tuning set), 51.4-54% on validation depending on whether duplicate validation images are excluded.
• 4 classes, 64x64, with proper split: Phase 2 in progress — expected to close the gap significantly since the HL loop will tune on train and report on val.
• 10+ classes, 128x128: likely much higher (resolution removes the saturation ceiling)
• 50 classes: unknown, the feature library has 5.7 classes per feature reuse on average, suggesting some room to grow
• 1000 classes: almost certainly requires hybrid (HL for structure + shallow NN for texture)

The experiment places a concrete stake in the ground: for a limited class space, Heuristic Learning achieves meaningful image classification, not toy-level, but legitimately competitive with neural baselines for the information available at this resolution.

The Coupling Complexity Picture

Weng defines coupling complexity as "the level of strategy complexity a coding agent can maintain." This experiment provides a concrete measurement:

The system: ~5000 lines of code, 40 features, 22 tiebreakers, ~50 interdependent thresholds across 10 classes.

The failure mode: by Session 11, every proposed fix risked regression somewhere. Adding a mushroom condition that catches 3 errors might flip 2 correct dogs. The coupling complexity was approaching the coding agent's maintenance capacity.

What kept it manageable:

• Modular features (each in its own file, independently testable)
• Full eval as instant regression test (230 images in <10s)
• Proof traces for failure diagnosis (no guessing why something failed)
• Experiment logs preserving what was tried and why it failed

This matches Weng's hypothesis: "Clearer feedback increases the coupling complexity that a fixed amount of agent intelligence can maintain." The proof traces and instant eval were the clear feedback that enabled maintaining a 50-threshold system.

Comparison with Weng's Results

This experiment extends HL into a new domain: static perception rather than sequential control. The HL loop still applies: feedback drives code changes that drive performance. But it also reveals a new phenomenon (representation saturation) that doesn't have an obvious analogue in control tasks.

Lessons for the HL Paradigm

1. HL works for perception, not just control

Weng's examples are all sequential decision-making (Atari, MuJoCo). This experiment shows HL applies to one-shot classification too. The "policy" is a scoring function, the "reward" is accuracy, and the "episode" is a single forward pass. The HL loop still works.

2. Representation saturation is the HL analogue of environment complexity

In control tasks, HL hits walls when the environment needs "composable macro-actions, recoverable search state, and long-term memory" (Montezuma). In perception tasks, HL hits walls when the signal lacks discriminative information. Both are cases where the current representation cannot absorb more feedback.

3. The coding agent's maintenance capacity is the real bottleneck

By Session 11, the system was at the edge of what targeted fixes could achieve without regressions. This is Weng's coupling complexity in action. The next step requires either a fundamentally new representation (higher resolution) or better tools for the agent (automatic regression isolation, feature independence testing).

4. Local corrections scale; global optimizations don't

Weng: "Rules that used to be one-off patches may start to become code worth owning for the long term." This experiment confirms it. Each tiebreaker condition is a "one-off patch" that permanently improves the system. The aggregate of 50+ such patches produced the final Phase 1 development-set result.

What This Means for the "Next Paradigm"

Weng asks whether HL could be the next paradigm after pretraining, RLHF, and large-scale RL/RLVR: "anything that can be continuously iterated on starts to become solvable."

This experiment provides a data point: yes, even image classification, the canonical neural network task, can be partially addressed by continuous heuristic iteration. The system reached its Phase 1 development-set result through pure HL, with no gradient descent anywhere in the loop.

But it also shows where HL alone isn't enough: when the signal lacks discriminative information, no amount of code editing helps. The hybrid future Weng suggests ("use HL to process online data quickly, turn online experience into trainable data, then periodically update the neural network") seems correct. For vision:

• Shallow NNs for texture features below the symbolic resolution limit
• HL for compositional reasoning, class hierarchy, tiebreaking, and interpretable scoring
• LLM agent for maintaining the HL system, proposing new features, and keeping the system growing

The division of labor is clear. Neither alone is sufficient. Together, they might be.

Technical Details

• Language: Python 3.11
• Dependencies: OpenCV, NumPy, SciPy (no ML frameworks)
• Inference time: ~25ms per image on M-series Mac
• Feature library: 40 registered features, 22 pairwise tiebreakers
• Lines of code: ~5000 total, ~3900 non-blank (classifier + features + sensors + eval)
• Development time: ~20 hours across 11 sessions
• Total eval runs: 248 (each testing a hypothesis)
• Coding agent: Claude (Anthropic), used for error-driven feature invention
• Estimated API cost: ~$100-300 in LLM inference (exact figure TBD, will update with actual billing)

References

[1] Weng, J. (2026). Learning Beyond Gradients. Blog post. https://trinkle23897.github.io/learning-beyond-gradients/

This experiment applies Heuristic Learning to static image classification, demonstrating that the HL paradigm extends beyond sequential control tasks. The system was developed iteratively by a coding agent (Claude) using the exact feedback loop Weng describes: environment feedback, read context, edit policy, rerun, write results, continue. The full codebase, 150+ evaluation logs, and reasoning traces are available in the repository.

Citation

If you reference this work, please cite:

Heuristic Learning for Image Classification: Without Neural Networks.
Xisen Wang, May 2026.