Jrsmyth's List: Canonical Model and Versioning

• One key aspect of SOA Governance that is often overlooked is how Data Governance can complement SOA Governance. Even though they both have very different objectives, they both share a set of metadata often called the “Enterprise Data Model”. An EDM is a Logical Data Model, an Ontology if you will, of the overall Information System. Its structure is often abstract and loosely related to the physical structure of the systems of record. However, all data elements stored in any given system of record should be traceable to an element in the EDM. The EDM is often used to construct maps to transform data being synchronized or replicated from one system to another.
• Message Types should be generated from EDM metadata.
• As Markus Voelter argued in his presentation, the main reason people have not used textual DSLs is simply because there were no simple ways to create parser and syntax sensitive editors (including intellisense). XText make it very simple to create the grammar and the rich editor experience that everyone expects (Figure 3).
• our approach does not require a “Canonical Schema Model” (CSM)

  • Not quite applicable to our use, but relevant comment
• SOA is not Integration,
• A CSM is also counter to many SOA Versioning principles as it “forces” all endpoints to conform to the same “canonical” schema, which we have seen is not practical since without the proper versioning strategy organizations often end up creating one service interface per consumer, thereby defeating the purpose of a CSM. SOA technologies (such as XML or WSDL) where actually designed to alleviate the need to always conform to a CSM which was a key design pattern for Integration Platforms in the 90s, but in the end this pattern propagated too many changes to the endpoints that did not need to change if only contracts could be designed in a compatible way.

4 more annotations...

• • discusses mapping from CDM to App model

• So just to clarify that maybe for users, using cookies is nothing special, but what you're getting at is that you don't use a cookie to store some kind of ID, which you then use on the server to look up session information data, which can be anything. You literally put all the state information into the cookie.

   

  Yeah, we do -- yes, with a few exceptions. So the few exceptions at the moment are for multi-page flows where -- so imagine somebody is selling an item. That's not something -- you don't enter all the information in a single page, but we need to retain that transient state going along. So how we do that is exactly how you hint, which is there is essentially an ID that's in the cookie that refers to something in a scratch database.

   

  In a database, right.

   

  Yeah, now, so it's not in memory or even on disk in the application server tier. It's physically on a database.

   

  That database is limited to one of your logical database things. So it's something that's kind of local to one of your applications.

   

  Yes, that's exactly right, and we are experimenting with other schemes where we have a session storage that's not in a database, but is separate from the application server tier, and we are kind of working the issues through on that. But we sort of think of those as server-side cookies, and just like cookies from the application developers' perspective, the developer has to assume that that cookie can be deleted and any time. So really important transient state like as for a multi-page flow, as I hinted, that goes in a database, as all our important information does. That doesn't live permanently in that database. That's why I call it a scratch database. But important data lives in databases and other transient stuff just gets passed back and forth. So that's our approach. And then the final example of partitioning is the search infrastructure, and that's the part that's near and dear to my heart because that's my particular area of expertise. So our search infrastructure -- so the functional segmentation is simply that the search infrastructure is read-only, as you might imagine a search infrastructure would be. It's functionally segmented from the rest of the transactional databases of the site, and I will talk a little bit about how the one gets transferred to the other --

• What gives? Are we always going to be haunted by the n-square problem? Not necessarily. We can apply the same hub-and-spoke concept to the issue of message formats, that is at the metadata level (metadata is the data that describes the format of messages). By introducing a "metadata hub" we can reduce the problem from n-square to a linear problem the same way as with the n-square physical connections. But what does a "metadata hub" look like? This time it is not a physical component, but a Canonical Domain Model that is common between all participants. When a message flows from participant A to participant B, the message format is first translated from A into the canonical format, and then from the canonical format into format B. It is sufficient to have a translation between each participant format and the canonical format, i.e. n translations for n participants (or 2*n if our transformations are unidirectional). Sometimes this concept is called "Semantic Hub-and-Spoke" but I generally prefer Canonical Domain Model to avoid confusion.

  • This doc talks through physical n-square to hub-and-spoke, then model n^2 to *. This repeats one more time at versioned sif/spi -