Rhm2ktmi's List: Application Architecture

• While Lewis and Fowler’s entire post is a fantastic read, one specific requirement stands out: microservices must favor “smart endpoints and dumb pipes.”
• Essentially, the smarts should be in the app components, and the communication between those components must be as simple as possible. Without that requirement, architectures will bloat and become monolithic.
• With smart pipes, the danger of increased complexity is high.
• Smart pipes create an unintended tight coupling between endpoints and the pipes themselves.
• Unfortunately, the temptation to make pipes smarter is high.
• Previous generations of developers introduced techniques like aspect-oriented prog
• Bootstrap policies are a modular vehicle for mixing cross-cutting concerns into endpoints on a PaaS.
• Endpoints can be smart while pipes remain dumb.

6 more annotations...

• First is vSphere 6, a big release many years in the making.
• One of the most exciting features for us in Cloud-Native
• First, Instant Clone can be used for traditional use cases where developers provision VMs to run fleets of containers.
• But a second and more interesting concept is that Instant Clone makes VMs so lightweight that we can potentially run one container per VM.
• Using Instant Clone for a one-container-per-VM model means you get VM-level security for containers, yet with instant provisioning and very low overhead.  It’s an exciting win-win and is a great example of the “containers without compromise” direction we took at VMworld last year.
• Finally, today we announced a new version of VMware Integrated OpenStack (VIO).  VIO is powerful because it provides the best of both worlds: the flexibility and API-driven nature of OpenStack with the maturity and power of VMware products. 

4 more annotations...

• Here are seven software architecture articles published in 2014 that can help you understand the basic topics and the current trends in software architecture: Agile, Cloud, SOA, Security… and even a little bit of data modeling.
• * Designing Software in a Distributed World
• * An SOA Maturity Ecosystem
• * Chop Onions Instead of Layers in Software Architecture
• * Security Architecture Approaches
• * Enterprise Data Modeling
• * Applying Test-Driven Development to Architecture to Keep Your Team on Target
• * Architecture Guidance for Microsoft’s Integration & Service Technology Portfolio

6 more annotations...

• This post is a rough “transcript” (with some changes and creative freedom) of a session I gave in the Citi Innovation Lab, TLV about how to effectively model a system.

• HashiCorp Consul is a solution for service discovery and configuration, designed to run distributed, highly available and scalable to thousands of nodes.
• Consul is used for service discovery, key/value configurations and health checking.
• There are also differences with more modern service discovery tools. SkyDNS provides a DNS interface to perform queries too, but the monitoring checks use heartbeating and TTLs, while Consul allows defining custom checks, as well as support for multiple datacenters out of the box.
• SmartStack, a project started by Airbnb, is a service discovery and registration framework composed by ZooKeeper, HAProxy and two Airbnb open source projects, Synapse and Nerve, used for service discovery and service registration.

2 more annotations...

• I advise teams to try and build out an internal PaaS capability — whether they are using Cloud Foundry or bootstrapping their own, or even several to allow for multiple application patterns.
• The Twelve-Factor App pattern is a good checklist of conditions to start with for understanding what’s necessary to get to a Heroku-like level of automation.
• What are the components of the application? Can you draw a graph of their dependencies?
• If the components are separated from one another, can they tolerate the partition or does the app crash or freeze? Are any components a single point of failure?
• How long does it take for the application to recover from a failure?
• Can the application recover from a typical failure automatically? If not what manual intervention is involved?
• How is the application deployed? If the server on which the application is running dies, what is the process/procedure for bringing it back to life?
• Can you easily replicate the state of your app in any environment?
• How do you scale the application?
• What does the Development/QA cycle look like?
• How does operations take delivery of code from development?
• How do you know that your application was successfully installed?

10 more annotations...

• Functional reactive programming (FRP) is a programming paradigm for reactive programming (asynchronous dataflow programming) using the building blocks of functional programming (e.g. map, reduce and filter). FRP has been used for programming graphical user interfaces (GUIs), robotics, and music, aiming to simplify these problems by explicitly modeling time

• The service promises developers the ability to focus on the projects they are doing and not the infrastructure that has historically been a constant consideration.
• Developing software as an aggregation of events that respond to change in data or state is not a new trend.
• Lately, event driven programming and related architecture has been positioned as the next evolution of SOA. Traditionally SOA has embraced a request-response model and is largely synchronous in its design. The event based model introduced asynchronous, push and pull, fire and forget methodologies. Most of the developers today are doing some form of event driven programming like setting onclick events in jQuery, AJAX callbacks etc.  So, this is not an entirely new idea to bend our head around.
• The advent of AWS in 2006 and later cloud native services, democratized distributed programming to the masses.
• A definition of a modern online application is now a collection of web services that persist their state outside itself, run independently on independent infrastructure, can be scaled horizontally and upgraded to avoid minimum or no downtime to the end user.
• Making reactive programming, event driven systems and functional thinking a new norm would mean a greater adoption of these ideas in the mainstream for enterprises and startups alike.

4 more annotations...

• That would be Amazon’s reveal of Lambda, which Amazon chief technology officer Werner Vogels simply described as “an event-driven computing service for dynamic applications.” It might sound like yet another service in the Amazon cloud’s ever-expanding portfolio. But really, it’s a tool to implement rules that carry out functions using Amazon’s manifold complex features, without requiring extensive configuration and maintenance.

   

• “This is revolutionary, or potentially revolutionary,” David Floyer, cofounder and chief technology officer of technology analysis firm Wikibon, told VentureBeat in an interview.

   
• On his blog High Scalability, developer Todd Hoff summarized Lambda as “PaaS++” — implying that Lambda is a souped up version of a platform-as-a-service (PaaS) cloud, which abstracts away the hardware underlying applications and merely executes lines of code. Even as he acknowledged its potential, Hoff had caveats. For the moment, Lambda has a limited reach, he wrote, given that it currently only supports the Node.js server-side JavaScript runtime and just a handful of events.

1 more annotation...

• Containers are a fast growing trend when it comes to delivering compute resources online.
• Each individual container posesses everything it will need from the operating system, file system, database, necessary libraries, and APIs to accomplish its given objective.
• Virtual containers will make API driven resources even more agile, nimble and scalable--not just for providers, but also for consumers.
• Think about when API resources become modular, customizable, scalable, wholesale resources that can be bought and sold, and deployed anywhere, making all digital assets into commodities that are ready for use anywhere in the API economy.

2 more annotations...

• An essential component of SOA is the interface, or contract. It defines the syntax of a service. It describes an interface by name, what information types the customer must supply when calling the service, and what a customer will get in return. The contract may also describe various service semantics utilizing comments included in the description or through the logical grouping of functionality - such as techniques or operations - into a common service unit. There's congruency between the SOA contract and the interface defined in languages like Java or C#, though unlike Java no semantic hints can be derived from static final variables, which are not exposed as part of a service definition.
• Although this seems like a contrived example, it actually illustrates an issue that is more prevalent than you may believe. Working within the context of an individual process space, there's a lot we get for free, so of course there's a great deal we take for granted. Security context (the state that declares who we are and what resources we can access), for instance, is maintained by the OS and calls to methods do not cross process boundaries so there are no integrity or privacy issues to be taken into account. Linkers make sure the code for a module is loaded and executed correctly as well as transparently. Distributed applications, in comparison, include few of these luxuries and a lot of new security dangers that need to be dealt with.
• Policy struggles with being an overloaded word. Within the discipline of IT security, it has a special multiplicity of definition. To a CSO for instance, policy is a set of corporate-specific security principles recorded in a binder. This may adhere to the conventional best practices spelled out in Internet RFCs 1244 and 2196, which explain how to formulate a comprehensive security policy for a company. As roles in a company come to be more finely grained, so too does the scope and interpretation of policy. The firewall administrator working in the CSO's business unit likely thinks about policies in terms of concrete rules applied to TCP and UDP ports. The HR department, on the other hand, defines policy regulating sick leave, carrying over vacation, and which staff members get a parking decal. Policy here usually doesn't have anything at all to do with security concerns.

1 more annotation...

• James Thomason, CTO, Dell Cloud Marketplace
• In the 10 or so startups I have helped build since 1995, one of the biggest challenges we always faced was the “scale problem”. Scale is a problem of success, a great problem you get to solve when you succeed at creating some initial value for your customers
• Today, software startups can build world-class web applications, scale them to millions of users, and ensure their availability for less money, time, and with employees than ever before.
• These advances in scale have not been without tradeoffs, however, and a new set of problems has emerged around development and operations complexity. While the horizontal scaling pattern increases capacity along with the number of application server instances, the difficulty becomes managing the sheer number of application server deployments, along with their configuration and dependencies on other infrastructure and services.
• One of the major benefits of Docker and containerization is that they enable an architectural style known as immutable infrastructure. In an immutable infrastructure, components of the infrastructure are never modified once deployed, but rather they are replaced by new components generated from a pristine state.
• To deal with the scale challenge, these steps would be automated through scripting (ssh for loop), configuration management (Puppet, Chef), and other external systems.
• In the new world of immutable application containers, applications are built in place with their dependencies and configuration.
• Containers are absent infrastructure dependencies, such as hostnames and IP addresses, which are injected at runtime when the container is deployed on a target host.
• This delineation provides a very clean separation of concerns between the application and the host infrastructure, and ultimately reduces the number of configuration endpoints by orders of magnitude (to just 1).
• The benefits for software development are vast.
• Already, Docker (and others) are working on various new forms of service discovery, in order to solve the infrastructure dependency injection problem, and consequently the “awareness” of dependencies between application components on different servers and infrastructure.
• This means that in the future, development teams can focus on creating and maintaining micro services, another important scaling pattern for software development process
• By leveraging the immutable architecture style, integration testing becomes a matter of deploying the necessary micro services from their pristine state in the repository and executing the test harnesses against the complete system.
• For operations, the immutable style provides a means to deploy major releases fractionally, testing them with small portions of the user base before rolling them out across the entire infrastructure.
• Containers themselves can be automatically introspected to derive dependencies on other containers, networks, storage, and other systems.
• In the last few months there has been a relentless outpouring of new orchestration systems for Docker containers
• Kubernetes (Google), Mesos (Mesosphere), as well as PaaS capabilities from Flynn.io, Deis, and OpenShift (RedHat)

15 more annotations...

•  
   
  <!-- .infoContainer -->  

  Shortened version of the Fast Delivery talk with a few new slides added since the GOTO Aarhus version earlier in the week..

• In the latest release of the ADC platform on which F5 Synthesis High Performance Services Fabric is built we've included both SPDY 1.3 and HTTP 2.0 support, enabling a gateway architectural approach to supporting the latest (soon to be) standard and the existing, more prominent one. This architectural feat is accomplished by way of BIG-IP's full proxy architecture, which lets our ADC speak one version a protocol on the outside (the client) and another on the inside (to the app).

    
• The performance and security (and let's not forget business) benefits to moving to HTTP 2.0 with its SSL/TLS requirements and improvements in core transport of data between client and server are worth exploring. But it's understandable that a protocol so entrenched like HTTP 1.x is not easily ripped out and replaced with something new.

• Shippable is a Seattle-based startup focused on the growing ecosystem and market around application containerization, primarily the open source Docker container. Shippable's software service helps enable faster, more efficient and more innovative application development and deployment by using containers in the application lifecycle and workflows
• Shippable is wise to get into Docker container development and management at an early stage – with Docker moving full-steam ahead as the basis of a rapidly growing and evolving ecosystem and market.

• oducts

  Shippable says its SaaS combines native container and Docker support with built-in enterprise roles and permissions. This means support for customer builds to run on containers that are provisioned on a multi-tenant build host with a 'share nothing' architecture, which means dedicated hosts for each customer and the ability to host them behind the firewall.
• The company claims its instant, containerized dev and test labs can cut an organization's cloud footprint in half by reducing testing and topology issues.
• The Docker ecosystem and market is still taking shape, but Shippable's primary competition comes from other Docker development and management tools. These include Panamax, backed by CenturyLink and Kubernetes, which includes involvement of Google and partners CoreOS, IBM, Mesosphere, Microsoft, Red Hat and SaltStack.
• Shippable is also competing with other PaaS providers that integrate with and support Docker, including ActiveState's Stackato, Amazon Beanstalk, Google App Engine, Pivotal, Red Hat OpenShift and WaveMaker
• Other Docker development and management tools that represent competition for Shippable are Cloudsoft with Clocker and New Relic with Centurion.
• Other competition comes from other test lab, continuous-delivery and devops vendors such as Electric Cloud, Plutora, QualiSystems and Skytap.
• Shippable faces some competition from open source developer PaaS options such as Cloud Foundry, Deis and Flynn, and open source Docker management projects such as Decking, Fleet and Shipyard.

7 more annotations...

• From its beginnings as a reliable storage pool with integrated batch processing using the scalable, parallelizable (though inherently sequential) MapReduce framework, we have witnessed the recent additions of real-time (interactive) components like Impala for interactive SQL queries and integration with Apache Solr as a search engine for free-form text exploration.
• Big Data architecture is premised on a skill set for developing reliable, scalable, completely automated data pipelines. That skill set requires profound knowledge of every layer in the stack, beginning with cluster design and spanning everything from Hadoop tuning to setting up the top chain responsible for processing the data.

• In the remainder of this post, you’ll learn about the various components in the stack and their role in creating data pipelines.

   
• Cluster Planning
• Ingress
• Staging
• Once the data has arrived in Hadoop as a whole, there remains the task of staging it for processing. This is not just about storing it somewhere, but rather storing it in the right format, with the right size, and the right access mask.
• Storage Formats
• Data Partitioning
• Access Control
• The last part you have to consider is what we call information architecture (IA), which addresses the need to lay out the data in such a way that multiple teams can work safely on a shared cluster—also referred to as multi-tenancy.
• It is not enough to have each job read from one directory and emit to another. If you share a cluster across departments, you need to devise a concise access schema that controls tightly (and possibly supports auditing of) who has access to what data.
• Data Processing
• Data Transformation
• Analytics
• Egress and Querying
• Data Pipelines
• Micro-pipelines are streamlined helpers that allow you to abstract (and therefore simplify) parts of the larger processing
• Tools for this purpose include Morphlines (see “Introducing Morphlines: The Easy Way to Build and Integrate ETL Apps for Hadoop” or details), Crunch, and Cascading.
• In contrast, Crunch and Cascading define an abstraction layer on top of the processing, where you deal with data points. You define how data is consumed, routed, processed, and emitted, which translates into one or more processing job on MapReduce and/or Spark. But a Crunch or Cascading “meta” job can further be combined to yet more complex workflows, which is usually done in macro-pipelines.
• Apache Oozie is one of those macro-pipelines tools.
• Productioniziation

20 more annotations...

• Contrary to a product roadmap, which expands along a horizontal time dimension, the data roadmap is defined by a vertical axis to indicate that data accumulates more value as it progresses through what I call the Five Stages of Data Transformation

• 1. Extraction

   

  2. Curation

   

  3. Derivation

   

  4. Combination

   

  5. Self Generation

• Recently microservices have been getting a lot of attention, both positive and negative. Articles about them tend to fall into two camps, which I'll affectionately label the hipster camp and the neckbeard camp. The hipster camp tends to strongly favor microservices, due to the excellent benefits they provide regarding separation of concerns, rigid interfaces, localization of data and the like. The neckbeard camp tends to be more suspicious, citing network latency, network unreliability, and the general fact that distributed systems are hard.