Orc is an orchestration language, as such it is designed to orchestrate operations or tasks. We will call these operations or tasks "sites"; however this does not imply anything about how they are implement or about how large they are. Orc needs to provide a way to define sites, such as importing it from another language or implementing it in Orc. How Orc interacts with sites also guides what features the Orc language should have and how complete of a language it should be. This issue of how to define a site and how big of a language Orc should be is the crux of the integration issue. Where do we draw the line between Orc and non-Orc? How to programmer interact with that line? '' The rest of this document discusses how sites are developed and integrated with Orc. In this case we will only concern ourselves with the language, not the implementation, so we will not worry about how the Orc implementation communicates with sites. '' !!! Site Integration Approaches There are two basic ways to integrate sites into Orc: external and embedded. External site integration is the equivalent of a foreign function interface in other languages. Some external code (such as a Java class) is referenced and invoked from within an Orc program. This approach is used by the standard Orc implementation. The site implementation is in a separate module (such as a separate file or project) than the Orc program. This means that development of sites and the Orc program that uses them is separate and should be conceptually distinct to prevent non-cohesive programming of a cohesive problem. Depending on features available in Orc, the ease of implementing new external sites may or may not be important. If Orc is rich enough, then any new sites can simply be implemented in Orc as an internal site instead in another language. However, integrating existing external libraries as sites is still important and should be easy. The techniques used to integrate the external sites does affect the programming experience in terms of what language integration work is done in and how automated it is. There are several obvious possibilities: # Using a unified communication model (such as CORBA, COM, ...), support for which is added to other systems. This causes integration code to be located with the library or system to be integrated and be written in its language. This approach can be automated if there is automated wrapping tools available for the library language. # Supporting multiple APIs and communication protocols in Orc. This places low-level integration burden (allowing simple calls into external code or APIs, and back from the external system into Orc) in the hands of the Orc implementers, but allows full fledged Orc level APIs to be implemented in Orc on top of this low-level interface. The Orc-level API could be autogenerated using stub generators. # Ad-hoc interfacing using a host language such as Scala. This means that every integration needs to be done in the implementation language of Orc. These three options are not totally distinct and overlap in some cases. Option 1 simplifies the Orc implementation since it does not have to support "non-native" integrations; however, this option also increases the work needed to integrate new libraries or systems with Orc since every system will need to support Orc's communication model. Option 2 moves some of the integration work into the Orc implementation (such as supporting low-level protocols, like gRPC or Java methods) thereby allowing many integrations to be done purely in Orc; there would still be integration code to write. Both options 1 and 2 could be simplified using automated code generators (or Orc compiler plugins of some kind). However, the generators would be specific to an external language or protocol so multiple would be needed. Option 3 forces all the integration coding into extensions to the Orc runtime; this options is likely to provide the best performance and the least work for the Orc implementers. However, it would also make integrating new systems the most complicated and error-prone. The current standard Orc implementation is a mix of all three approaches. It supports simple calls into Java allowing some Orc wrappers to be written in Orc, but due to limitations on the Java import support some things (such as GUI or web framework callbacks) cannot be implemented in Orc and require some custom stub code in Java or Scala. In addition, Orc provides a Java level API which allows Java or Scala code to fully interact with Orc programs, however the Java or Scala code must be implemented with Orc in mind. Embedded site integration allows sites to be defined where they are used by providing code in some other language embedded in the Orc program. Unlike an external site, the code for an embedded site appears where it is called. This approach is supported by Orclaves using "Scalaclaves" which are effectively embedded sites. This allows sites to be cohesively developed with their caller, but also introduces semantic variation to the language since the site implementation code is not Orc. An important choice with embedded is which language to support for embedded site implementations. Orclaves are an embedded DSL in Scala so the embedded site language is also Scala, however multiple embedded languages could be supported using a compiler which separates an Orc source file into multiple units each compiled by a separate backend and then compiling the result on a common runtime (such as the JVM). In the case of embedded sites, any access to external libraries can be done from the embedded language, so no special support is needed to embed libraries written in or supported in one of the embedded languages. !!! Integrating Orc with Other Code __TODO:__ This issue is directly connected to the above. The embedding in both directions is coupled. No reason to do one without the other. An issue which was have not addressed is what kind of unit is an Orc program in a larger system. The standard Orc implementation makes Orc the root of the program invoking other systems but not easily invoked by them. For instance, it would be challenging to have a Scala program which uses Orc for an object or two or for one method on an object. On the other hand, Orclaves are fundamentally designed to embed Orc programs within Scala programs in nearly any position in the program. Similarly, Orclave objects could allow single objects to be written in Orc without effecting other objects in the system. The same embedded vs. external choice appears in this direction too. Embedding Orc would allow Orc code to be co-developed with host language code more easily. However, keeping Orc external would simplify code understanding in some cases and would make the boundaries more clear. This issue interacts with the Orc language design in that if Orc is used to implement methods/functions called from some other language there is much less need for an modularity constructs in Orc since each "Orc program" will only need to implement one operation, not an entire system. On the other hand, if Orc is expected to be the entry point to start a system then it needs to have the features to support a larger set of interactions than a single method/function. !!! Orc Language Design The Orc language could come in one of several forms. Here we will look at two: # Orc as a orchestration DSL not designed for any other tasks. # Orc as a complete language in which whole applications are developed. An Orc DSL has the advantage that there is no need to extend Orc to provide all the modularity and large scale structure features needed by larger programs. Similarly, the Orc type system or other static analysis tools would not need to support these more advanced features and therefore could be kept simpler. Notably, an Orc DSL could still be embedded in and embed sites from a more sophisticated language with advanced features. The other language's features could be used to structure Orc programs, especially if Orc is embedded in that language. !!! Combining These Choices These three axes are far from orthogonal and it is not yet clear what the interactions between these choices are. However, the current implementations of Orc cover a surprisingly wide range of this space and provide some light on what the results are. * The released Orc 2.1 implementation provides a relatively small Orc language designed for writing relatively small functional programs which orchestrate large sophisticated sites. In the terms from above, this implementation uses a simple language with external sites and external invocation from other programs. * The development Orc "3.0" implementation with [objects|OrcO] provides a much larger language designed to allow developing much larger programs using simpler sites. In the terms from above, this is a sophisticated language with external sites and external invocation from other programs. * [Orclave] provides an Orc implementation entirely embedded in another language with sites replaced by fragments of Scala code embedded directly in the Orclave code. In the terms from above, this is a simple language which is embedded in and embeds another language (Scala). These different design points are useful for different styles of programming. For instance, Orclaves allow otherwise Scala-based programs to use Orc's capabilities natively. However, an application which integrates systems written in many different languages would not be well served by Orclaves since Orclaves rely on Scala (limited) support for integration with external languages. If the main non-Orc language is not Scala or there is no single external language at all, then a stand-alone Orc language with it's own external integration features would serve the programmer better.