Table of Contents

Extension Compatibility

Note

This is a historical design document and may be outdated.

Problem statement

In instagile framework release 222.0, calculated properties are now modelled. The model editor is used to set calculation modes (instead of code attributes) and to configure what kind of abstract calculation functions, if any, are generated. Because a specific version of The.Model.WPFEditor is bundled with Instagile.vsix, using the latest version of the extension will give you the version 222 UI, which is bad for editing .theModel files in applications which have not updated to runtime 222. Any edits you make to the calculation mode or display format would generate model events that wouldn't be understood by the old runtime.

When the editor UI has changed in the past we've updated runtimes & extension versions together, sometimes using different extension versions to work on different projects. Martin does not want to repeat that situation and wants this model change to be made forward-compatible, so that a newer version of the editor can correctly edit a model file for an older version of the runtime. It's not required to solve the forward-compatibility problem for the general case, just for this particular change - in practice, most editor changes already do not interfere with editing old model files.

Background

The canonical representation of an app schema (its "model") is a series of model events, which are defined by metadata in Events.t4. As an object representation, there's no inherent cross-version compatibility - in either direction. Within a loaded assembly, there is only one possible version of the metamodel. This restriction gets us important benefits: lots of t4-generated code for model manipulation, performance benefits from immutable snapshot types, event stream optimisation, the use of events as dispatch-actions in the editor's architecture, consistent hashing for license checks, test suites which are statically comprehensive, etc.

It isn't feasible to change the concept of model events per se, but obviously having no backward- or forward-compatibility would make the architecture useless. Since the beginning, we've implemented backwards compatbility using versioned serialisation. Each event has a version number in its type - ChangeEntityV5, ChangeEntityV6, etc. theModel files contain an overall model version which is used to interpret the events within. The model-loading library migrates obsolete events to updated ones when loading a model from an older file version, and file version N will sometimes make larger changes like dropping support for some old model element or introducing a new concept like snapshots. We're currently on version 7 of the ModelFile object, and in general the process for updating versions has been that you open an old theModel with a new editor, save it, and it migrates events as necessary.

This scheme works well for backward-compatibility, but doesn't address forward-compatibility. This became a serious problem over time; when running code generation meant loading a solution's theModel into the autoupdating extension, too-new an extension version would be unable to execute generation for older templates. Almost any change to the in-memory model representation, either events or snapshot, would break, and you had to match an extension version to a runtime version within a pretty narrow range. We eventually addressed that situation by decoupling generation entirely from the editor; The.Extension communicates over IPC with an isolated exe, The.Extension.Generator, which loads the runtime's version of the model events through a stable, minimal reflective interface.

Decoupled generation had no benefit for editing, but due to the nature of model events this has rarely been a problem. Most event properties are optional, so if the editor has a new element which your runtime doesn't support, you simply don't use that element. Components/'model features' presented an issue, because they have definitions supplied by the model which might evolve over time; Jeff solved this problem by introducing a new versioned element to theModels. It describes the versions of features (like 'user profiles') known to be present in a model, and achieves forward-compatibility by calculating the feature version from a model's events - a 'feature' being just essentially just a preset series of events that add things to the schema.

Challenge: Version detection

In order to do anything differently, the editor needs to know that it shouldn't use the new calculated-properties paradigm. This poses a problem because there's no channel of communication from the runtime to the editor. Editors produce a model, which is consumed by the runtime. Furthermore, there's nothing in existing model files which could communicate this information - every app does not currently have modelled calculated properties. There's no obvious general solution for the problem of "for what runtime is a .theModel intended", but if we don't attempt to solve for the general case, we can use one or another form of out-of-band information. Here are a few ideas, from most to least promising:

  1. The model editor doesn't know anything about runtimes, but the extension which hosts it does. The.Extension's IPC mechanism can be used to query the runtime, if one is present in the solution; I've actually already written the RuntimeInformation type which does this, acquiring minimum- and maximum- supported ModelFile versions. We don't use that for anything, and max model version might or might not be the appropriate way to communicate this feature support, but if not it can be extended to communicate actual runtime version.
  2. The editor has settings which are persisted into the VS registry hive. Users could manually toggle the editor between modes when working on an old/new model file. Obviously the problem here is that you'd have to keep switching it back and forth - it's like switching extension versions, but considerably easier.
  3. The FeatureVersions mechanism does something similar, but not identical, to what we need. It tracks the level of support within a model for a specific IFeature; this could be broadened and reconceptualised as tracking model concepts. There's a risk of greatly complicating things, however, and it doesn't solve the problem of deducing whether a file with no FeatureVersion should support modelled calc props - at minimum the user would have to click some sort of "enable calcprop modelling button from now on" button.

Challenge: Adaptive UI

The editor UI has three parts: a viewmodel node tree, a WPF data-binding view and a Blazor data-binding view. Adding this capability means a new kind of communication from the node layer upwards - it must specify flags for which UI layout to use, and provide the backends for both. Each view must then check the flag and render slightly different UI, data-binding to different subsets of the underlying properties. This isn't conceptually super complicated, it's just a special case, and every one of these forward-compatibility implementations would be another special case. If we had a lot of them it would be a complex compatibility matrix, so it makes it harder to change this area of the editor in future - unless pre-calcmodes models become obsolete, which we could do using the MinVersion mechanism eventually.

Actually updating the viewmodels is done by processing an event sequence, which can come from many sources - loaded from a file directly, generated by the user's input gestures, modified by the optimiser, undo/redo, etc. All the editor controller will need to do is apply both possible meanings of events. If it sees ChangeAttribute:IsCalculated=false, it will need to interpret that both as a boolean element and as a setting for the new three-way dropdown. The event itself will be the same regardless of what the current attached runtime, if any, understands. Note that this technique relies on the 'canonical latest version only' nature of the model events!

Challenge: Generating backward-compatible serialisations

Once you've got reliable min/max version detection, this is the tricky bit. Fundamentally, the model editor is dealing with a single version of the model events - the latest - but those events can themselves be versioned and parts of the events can be unused. Let's assume that there will be a new model version, 8, which adds some new event versions or changes the meanings of some events; the new trick the editor then needs to learn is to generate different events, either older versions or different content, based on the compatibility level it's trying to achieve. Using redundant property elements on the tree nodes should let us achieve this; the v7 properties will generate v7 change events.

Unfortunately, creating new events is done all over the editor (and the rest of the model code, optimiser snapshotbuilder etc). This will need to be audited, both for cases where the specific events we care about are being created and for cases of cloning/modifying entire model elements - for example, copying an attribute involves creating a CreateAttribute event that sets all the properties of the new attribute to those of the old. If the MaxVersion information is threaded through all of these model manipulations, then it ought to be sufficient to support this case and also future instances of special foward-compatibility UI. The model-saving code will then need to be redesigned a little to allow for deliberately not saving the latest version - having taken all that care to preserve a file at V7 instead of V8, we can't just let the normal upgrade-on-save process go ahead

Estimate

By using one of the existing partially-appropriate mechanisms for version-detection we could definitely get the information the editor needs - as long as we can target this one specific case, not solving the general problem. That's only a day or two of work. Duplicating the relevant node-tree properties and applying multiple meanings is tedious, but trivial; again 'a day or two' and hopefully they fold into each other. Generating backward-compatible events is where the unknowns lie. There are assumptions all throughout the modelling code that the 'latest version' of events is always correct to create. I see no conceptual reason we can't change this in each instance, but it might be difficult to get them all, or to pass the version-range restrictions through to where they're needed.

There are also miscellaneous logistics: we probably have to declare 222.0 a dead release and make 222.x/223 require a new model version, rather than interpreting the existing events differently.

To get all this right and release an updated 222.x, I would expect to spend the rest of the week on this work. It's unlikely to blow out but also not likely to go significantly faster. What we'd get out of it is forward- and backward-compatibility for this specific change, as well as little bits of infrastructure and understanding toward being able to make similar changes forward-compatible in future.

Alternatives

  1. We could live with the problem, updating applications quickly or keeping old extension versions installed when that's not feasible.
  2. We could attempt a more comprehensive approach by running the editor UI itself out of the runtime, much as we do with the generation code. This would solve the whole class of problem but comes with enormous challenges of its own (shipping the model editor with apps! can visual studio hosting be made to work at all?! etc).