client-training.md

Client Training Mechanism

Summary

This issue covers how client-side local training should work in DevNet and why the current implementation is too limited.

Today the client service assumes:

• each client has a local dataset
• that dataset is labeled
• local training is standard supervised learning
• a single training loop with CrossEntropyLoss is sufficient

That is acceptable for a demo such as MNIST, but it is not a robust general federated learning design.

Current State

Current implementation is centered around:

• cache_model_0/services/client.py
• model manifests that expose train_client_model_and_upload_to_ipfs

The current flow is roughly:

1. Client retrieves the genesis model.
2. Client loads local dataset from dataset/clients/<account_address>/data.pt.
3. Client optionally initializes from the previous global model.
4. Client trains locally with a PyTorch DataLoader.
5. Training uses CrossEntropyLoss and Adam.
6. The trained local model is saved and uploaded to IPFS.
7. Optional differential privacy noise is added after training.

Important current details:

• the training loop expects batches shaped like (inputs, labels)
• the learning objective is supervised classification
• there is no explicit handling for unlabeled data
• there is no label-quality validation
• there is no training-policy abstraction beyond a single hardcoded loop

Why The Current Design Is Weak

1. It assumes labels exist locally

That is a strong assumption.

In many realistic FL settings:

• clients have raw inputs but no labels
• labels are delayed, incomplete, or expensive
• labels may be generated by heuristics instead of humans
• some clients have labels and others do not

The current training path cannot operate in those cases.

2. Supervised gradient descent is not the only training mode

The question is not "can gradient descent work without labels?"

Gradient descent can still work without labels, but not with the current supervised loss. Training without labels requires a different objective, for example:

• self-supervised learning
• contrastive learning
• masked prediction
• autoencoding or reconstruction losses
• pseudo-labeling
• distillation from a teacher model

So the real issue is that the client service hardcodes one objective rather than supporting multiple training regimes.

3. Local labels may not be trustworthy

Even if labels exist, they may be:

• noisy
• stale
• weakly inferred
• adversarially manipulated
• inconsistent across clients

If the protocol treats all client labels as equally trustworthy, local training can amplify bad data and poison the global model.

4. Client data heterogeneity is not addressed

Federated learning usually has:

• non-IID data
• varying dataset sizes
• skewed class distributions
• uneven device capability

A single fixed training loop with fixed epochs and fixed optimizer settings is unlikely to be reliable across clients.

5. There is no explicit training-policy contract

Right now the service interface says "train and upload a model", but not:

• what kind of training is being performed
• what assumptions it makes about labels
• how to handle clients with partial labels
• how to verify that the local objective matches protocol expectations

That makes the system hard to generalize.

Core Questions This Issue Should Answer

The protocol needs a clear answer to each of these:

1. What training modes are supported?
2. What happens if a client has no labels?
3. How are labels obtained or inferred?
4. How much does the protocol trust local labels?
5. What checks exist against noisy or malicious local training?
6. How should different training modes affect scoring and aggregation?

What Kinds Of Models Can This FL Stack Support?

At a high level, the DevNet stack is more flexible than the current MNIST-style demo suggests.

The customization path is roughly:

1. task-level contracts
2. manifest
3. services
4. dincli behavior that relies on the manifest

In practice, that means the protocol is mostly driven by service logic and manifest wiring, while contracts provide the coordination and settlement layer.

Current Flexibility Boundary

The current contracts mostly coordinate:

• submission of local model artifacts
• auditor assignment and scoring
• aggregation approval
• reward and slashing flow

That makes them relatively model-agnostic as long as:

• a client can serialize a local update or model artifact
• aggregators know how to combine it
• auditors know how to evaluate it
• the manifest points each stakeholder to the correct custom services

So yes, the stack is flexible enough in principle to support many FL model types through customized services.

But "any model" is only true if the surrounding training, aggregation, and scoring logic is also customized appropriately.

Model Families That Can Be Supported

Examples include:

• classification models
• regression models
• sequence models
• recommender models
• ranking models
• anomaly detection models
• representation-learning models
• multimodal models
• decision trees
• tree ensembles such as random forests or boosting variants
• linear models
• clustering or prototype-based models
• LLMS
• FedLORA

The key point is that federated learning is not limited to neural-network classification.

Frameworks That Can Be Supported

PyTorch is the current default in the demo code, but the architecture does not need to be PyTorch-only.

With customized services, the stack could support:

• PyTorch
• TensorFlow or Keras
• JAX
• scikit-learn style models
• XGBoost or LightGBM style tree systems
• custom native or serialized model formats

dincli does not need to understand every framework natively if the manifest-defined services handle:

• loading the model
• training the model
• serializing the output artifact
• aggregating updates
• evaluating submissions

Where The Real Constraints Are

The real constraint is not only the framework. It is whether the FL method for that model family is well defined.

For example:

• neural networks usually aggregate parameter deltas or weights
• linear models may aggregate coefficients or gradients
• decision trees do not fit simple FedAvg-style averaging
• boosted trees may need round-based structure sharing or ensemble merging
• personalized models may share only part of the model

So different model families may require different:

• client training logic
• update format
• aggregator logic
• auditor scoring logic
• eligibility checks

What Contracts Need To Assume

Task-level contracts should stay as generic as possible.

They should avoid assuming:

• one ML framework
• one model architecture type
• one update format
• one scalar utility interpretation

Instead, contracts should mostly assume that off-chain services produce:

• a submitted artifact CID
• whatever metadata the workflow requires
• compact score or approval outputs that can be finalized on-chain

That is the cleanest way to keep contracts reusable across model families.

What The Manifest Should Control

The manifest is the natural place to declare model-specific behavior.

It should define or point to:

• model type
• framework
• client training service
• auditor service
• aggregator service
• dataset assumptions
• label assumptions
• evaluation mode
• optional metadata schema

This is the layer that tells the rest of the stack what kind of FL task is being run.

What Services Must Customize

Services are where true model flexibility lives.

To support a non-default model family, custom services may need to define:

• how the model is loaded
• how local training works
• what artifact is uploaded
• how aggregation works
• how auditors evaluate utility
• what counts as a malformed or invalid submission

This is especially important for non-neural models such as:

• decision trees
• boosted ensembles
• probabilistic models
• retrieval systems

Those often need custom aggregation and custom scoring, not just a different training loop.

Recommended Position

The right protocol position is:

• contracts should remain generic and coordination-focused
• manifests should declare model and workflow semantics
• services should implement model-specific FL logic
• dincli should orchestrate the workflow from manifest-defined services

With that structure, DevNet can support a wide range of FL models and frameworks.

But support should be described as "customizable for many model families" rather than "automatically supports any model" because some families require fundamentally different aggregation and evaluation semantics.

What If There Are No Local Labels?

This is the central design question.

If there are no labels, the current supervised client loop cannot run as written because CrossEntropyLoss needs targets.

But that does not mean local training is impossible.

It means the protocol must support alternate objectives.

Viable Training Modes Without Labels

Option 1. Self-supervised pretraining

Clients train representations using unlabeled local data with objectives such as:

• contrastive learning
• masked modeling
• next-step or next-token prediction
• reconstruction losses

Pros:

• works without labels
• often fits FL settings well
• useful when unlabeled data is abundant

Cons:

• harder to evaluate directly
• may not align perfectly with downstream task quality

Option 2. Semi-supervised learning

Some clients have a few labels, most data is unlabeled.

Typical approaches:

• supervised loss on labeled samples
• consistency regularization on unlabeled samples
• pseudo-labeling with confidence thresholds
• teacher-student training

Pros:

• practical for real deployments
• uses both scarce labels and abundant unlabeled data

Cons:

• quality depends heavily on pseudo-label quality
• can reinforce early model bias

Option 3. Weak supervision

Labels are derived from heuristics, rules, external models, or user events.

Examples:

• rule-based labeling
• programmatic labeling
• click or interaction signals
• metadata-derived labels

Pros:

• often cheaper than manual labeling
• realistic in many domains

Cons:

• noisy by design
• requires confidence modeling and label auditing

Option 4. Distillation from a trusted teacher

A global model, model-owner model, or foundation model can provide soft targets.

Pros:

• useful when human labels are missing
• can align local training with a shared target

Cons:

• imports teacher bias
• requires trust in the teacher
• may centralize too much control

Option 5. Personalization-first training

Clients optimize for local utility while sharing only part of the model or a common backbone.

Pros:

• strong for heterogeneous data
• more realistic when clients have different tasks or label semantics

Cons:

• harder to aggregate
• harder to compare client contributions directly

Are Local Labels Trustworthy?

Not automatically.

The protocol should assume local labels can vary in quality and that some clients may be careless or adversarial.

Label trust should be treated as a spectrum:

High-trust labels

• obtained from verified human annotation
• stable ontology or schema
• audited data collection process

Medium-trust labels

• operational labels from product events
• labels produced by internal workflows
• delayed or partially noisy human review

Low-trust labels

• heuristic labels
• self-reported labels
• automatically scraped labels
• adversarial or unverifiable labels

Training policy should depend on label trust level, not just label presence.

How Can Local Labels Be Achieved?

When raw local data has no labels, labels can be introduced through several paths:

A. Human annotation at the edge

Clients label their own local data.

Pros:

• direct task alignment

Cons:

• expensive
• inconsistent
• hard to verify

B. Centralized labeling guidelines and schema

The model owner defines strict label semantics and tooling.

Pros:

• better consistency across clients

Cons:

• still depends on execution quality

C. Weak labeling functions

Use domain rules or heuristics to assign labels.

Pros:

• scalable

Cons:

• noisy

D. Pseudo-labeling from a global model or teacher

Use model predictions as labels for confident examples.

Pros:

• no human labeling required

Cons:

• prediction errors become training data

E. Task redesign

In some cases the better answer is to redefine the task around signals that naturally exist, such as:

• next-item prediction
• anomaly detection
• ranking from interaction logs
• representation learning for downstream fine-tuning

This is often more honest than pretending missing labels can be solved cheaply.

Recommended Training Direction

The protocol should stop treating client training as one hardcoded supervised loop and move toward a policy-driven training stack.

Proposed Training Stack

Layer 1: Dataset and label profile

Every task should declare:

• labeled, unlabeled, or mixed data
• label trust level
• expected input format
• task type
• expected loss families

The client service should train according to this declared profile.

Layer 2: Training mode selection

Supported modes should eventually include:

• supervised
• semi_supervised
• self_supervised
• weak_supervision
• distillation
• personalized

This selection should be explicit in the manifest or generated service artifacts.

Layer 3: Mode-specific training logic

Each mode should define:

• required dataset fields
• loss functions
• optimizer defaults
• stopping criteria
• confidence filters
• update clipping or regularization rules

Layer 4: Reliability checks

Before accepting local updates, the protocol should consider checks such as:

• update norm bounds
• loss explosion detection
• NaN checks
• training metadata completeness
• divergence from cohort updates
• consistency with declared training mode

These checks do not prove training quality, but they reduce obvious failure modes.

Robust Training Approaches Worth Supporting

1. Stronger supervised baseline

Even for labeled tasks, the current baseline should improve:

• configurable optimizer and learning-rate schedule
• configurable epoch and batch policy
• class imbalance handling
• validation split support
• early stopping
• per-client metadata reporting

2. Semi-supervised FL

This is likely the best near-term expansion because many real clients have mixed labeled and unlabeled data.

Good candidates:

• FixMatch-style consistency training
• pseudo-labeling with confidence thresholds
• Mean Teacher style approaches

3. Self-supervised representation learning

This is the cleanest path when labels do not exist at all.

The protocol can train shared representations first, then:

• fine-tune with labels where available
• evaluate downstream tasks separately

4. Distillation-based local adaptation

Useful when a trusted teacher or global reference model exists.

This can reduce dependence on local labels while keeping clients aligned to a shared task.

5. Personalized FL

Useful when client distributions differ too much for one global model.

Examples:

• shared backbone plus local head
• regularized local fine-tuning
• clustered or subgroup federated learning

This is more robust than forcing one global objective on incompatible clients.

Honest Design Boundaries

The protocol should be explicit about what it can and cannot claim.

If there are no labels anywhere

Then the system can still do:

• self-supervised learning
• representation learning
• anomaly detection
• pretraining

But it cannot honestly claim supervised task performance improvement unless some downstream labeled evaluation exists.

If labels exist but are low-trust

Then the system can still train, but:

• scoring should discount uncertain labels
• aggregation should be more robust
• auditing should include label-quality assumptions

If clients use different label semantics

Then standard global aggregation may be invalid.

The protocol may need:

• schema alignment
• ontology mapping
• client clustering
• personalized training paths

Relationship To Auditor Scoring

Client training policy and auditor scoring cannot be designed independently.

Examples:

• if clients train with self-supervised objectives, auditors should not score only top-1 classification accuracy
• if clients use pseudo-labels, auditors should consider confidence and label provenance
• if training is personalized, aggregation and scoring should not assume one universal metric

Training mode must inform scoring mode.

Contract And Service Implications

This issue is mostly about service and workflow design, but it has protocol implications.

Likely changes:

Client service

train_client_model_and_upload_to_ipfs(...) should become policy-driven rather than assuming:

• one dataset format
• one loss
• one optimizer
• one fully supervised objective

It should support richer task configuration and emit training metadata.

Manifest and generation workflow

Model manifests or dincli generation should declare:

• task type
• training mode
• label availability
• label trust level
• expected metrics
• safety constraints

Audit and aggregation

Auditing and aggregation should consume those declared choices instead of assuming every client update came from identical supervised training.

Recommended Policy

For DevNet, the recommended direction is:

1. Keep supervised labeled training as the baseline demo path.
2. Explicitly document that it assumes trustworthy local labels.
3. Add training-mode selection to the client service contract.
4. Prioritize semi-supervised and self-supervised modes next.
5. Add metadata and safety checks around local updates.
6. Align auditor scoring and aggregation policy with the chosen training mode.

Proposed Scope Of Work

Contributors working on this issue should aim to:

• document supported client training modes
• define dataset contracts for labeled and unlabeled tasks
• define label trust categories and their protocol implications
• replace the single hardcoded supervised loop with configurable training policy
• add support for unlabeled or mixed-label local datasets
• define required training metadata for auditors and aggregators
• add safeguards against bad local training and noisy labels
• connect training mode selection to evaluation and reward logic

Bottom Line

The current client service is a narrow supervised demo path.

The protocol should move from:

• one labeled dataset format
• one classification loss
• one local training loop

toward:

• explicit training modes
• honest handling of unlabeled data
• explicit assumptions about label trust
• more robust and reliable client-side learning strategies