Dyalog Version 16.0

Summer is here, and as usual that means that the next release of Dyalog APL is about to appear. We expect to make version 16.0 generally available on Microsoft Windows at the end of June 2017. IBM AIX, Apple macOS, and x86/x64 and ARM-based Linux will follow in quick succession.

Performance

Our work on rewriting core components in order to take advantage of modern processor features continues:

v14.0 to v14.1: 5%	v14.1 to v15.0: 24.9%	v15.0 to v16.0: 6%

The above charts show the performance profile of each of the last three releases over the preceding one (blue represents expressions that run faster). Our standard performance test suite reports that v16.0 is about 6% faster than v15.0. The last three development cycles have accumulated a speed-up of approximately one third, according to these (admittedly synthetic) tests. Our test suite runs about 14,000 expressions, applying functions to big, small, square, fat and tall arrays of various data types. Your mileage will vary: if you have performance critical code that you can share, please send it to us and we'll be very happy to include it in our tests and use it to direct future efforts.

Importing and Exporting Data

We know that the data volumes that applications need to handle keep growing. Therefore, in addition to the general speed-ups mentioned above, we are adding features to help you move data efficiently in and out of your application. Version 16.0 adds a high-performance mechanism to convert data held in CSV files into arrays and back.

The new system function ⎕CSV has defaults that make it very easy to get started, automatically detecting the characteristics of most CSV files, and it also has a plethora of options to allow you to decide how to deal with quoted or non-quoted strings, different separator characters and decimal markers, validate that columns are numeric on input and convert the data. You can choose between rectangular matrices or "inverted" arrays where each column is represented as a simple enclosed array. Despite the name, ⎕CSV also deals with fixed record layouts without separators at all – and provides very significant speed-ups over current best practices.

Exchanging Data with Other Applications

Data isn't always exchanged via files: often you want to call – or be called by – other applications over a network. Our TCP library Conga has been significantly beefed up in version 16.0. We have removed the burden of dis-assembling HTTP packets from the APL developer by supporting the HTTP protocol at the library level – which also allows us to add support for asynchronous, bi-directional WebSockets, for highly interactive web apps. Last but not least, Conga has also been optimised to better support very large numbers of simultaneous connections.

The experimental features for converting messages in JSON format to APL arrays or namespaces have been promoted into a ⎕JSON system function, complementing the existing ⎕XML function.

Cross-Platform Development Environment

The screenshot above shows a prototype of version 4.0 of our Remote Integrated Development Environment (RIDE). With Dyalog version 16.0, RIDE is promoted to be the default IDE for not only macOS (as it was with v15.0), but also Linux – including the Raspberry Pi. In addition, a "zero footprint" option is available, allowing you run RIDE in any web browser that supports Javascript, loading it directly from a running APL interpreter.

RIDE 4.0 supports many new features that were previously only available in our native Windows IDE, including a Workspace Explorer tree view, Stack and Thread windows and full support for debugging threaded applications.

Cross-Platform User Interfaces

The Graphical User Interface tools built-in to Dyalog APL are based on the Win32 API, which Microsoft introduced as part of Windows NT in 1993. Although this API will continue to be available for a long while yet, we have been searching for a technology that will last for the next 25 years – and be available on all the platforms where we expect Dyalog APL to appear. We now believe that HTML and JavaScript will provide the solution that we are looking for.

Version 16.0 includes a new HTMLRenderer object, which is available on all platforms except AIX (because APL for AIX is only ever used as a server environment). The HTMLRenderer allows APL applications to create user interfaces by generating and "injecting" HTML/JS into an instance of the Chromium Embedded Framework, which is included with Dyalog APL. You can also mix content created by your APL process with services provided by external web servers and use simple call-back functions to respond to user actions.

You can generate content using your own code, or you can use tools like SharpPlot, as in the above example which is using the HTMLRenderer to display a contour plot under macOS. You can use the MiServer widget library to generate rich content from APL, using all the controls that are on display at http://miserver.dyalog.com. No matter how you generate the content, the UI will look and behave in exactly the same way on all platforms – and the application code will be able to run as an actual web server if the application code support multiple users.

A New Approach to Tools and Samples

To make it possible to use industry standard source code management tools and other DevOps infrastructure, Dyalog has added support for APL source code stored in Unicode text files as an alternative to workspace files. Amongst other things, this makes it possible to share open source APL projects using tools like GitHub, allowing users to collaborate on developing tools and applications and to read APL code without having a running APL system.

With version 16.0, we are taking advantage of the new capabilities in the distribution of Dyalog APL itself. For the first time, an initially small batch of new tools and samples are included in the form of text files, rather than embedded within workspaces. Under the main Dyalog folder you will find a handful of new folders containing source files. These folders are all on the default path that is searched by the ]Load user command, which makes them easy to load. For example:

      ]Load HttpCommand
      site←'http://hasthelargehadroncolliderdestroyedtheworldyet.com'
      t←HttpCommand.Get site,'/atom.xml' ⍝ retrieve contents of URL
      {(2=⍵[;1])⌿⍵[;2 3]}(⎕XML t.Data) ⍝ leaves at level 2 of nesting
 title      Has the Large Hadron Collider destroyed the world yet?
 published  2011-08-08T15:23:48Z
 updated    2011-08-08T20:39:20Z
 content    NOPE.

Each of the new folders is a snapshot of a corresponding open source repository located on GitHub, taken just before the release of the new version. These repositories will be continuously updated and have their own release cycles, visible to all via GitHub. Contributions are welcome!

New Primitives

Version 16.0 puts no fewer than four new glyphs (⍸, @, ⌺ and ⊆) into use, providing significant new functionality at the core language level:

      ⍸ 0 1 1 0 1
2 3 5

      2 5 10 ⍸ 1 2 3 4 5 6
0 1 1 1 2 3

      ((⊂'fizz')@{0=3|⍵})⍳6
┌─┬─┬────┬─┬─┬────┐
│1│2│fizz│4│5│fizz│
└─┴─┴────┴─┴─┴────┘
      upcase←1∘(819⌶)
      (upcase@1)¨'hi' 'mom'
┌──┬───┐
│Hi│Mom│
└──┴───┘

  [argument] (modifier @ selector) array

      data←0 0 8 0 0 6 0 0
      filter←0.25 .5 .25
      blur←{+/filter×⍵}⌺3
      ↑data (blur data)
0 0 8 0 0   6 0   0
0 2 4 2 1.5 3 1.5 0

      ⊆ 1 2 3
┌─────┐
│1 2 3│
└─────┘

      1 1 2 2 2 ⊆ 1 2 3 4 5
┌───┬─────┐
│1 2│3 4 5│
└───┴─────┘

Conclusion

In addition to improved performance, import/export tools, new communication tools and cross-platform features, Dyalog APL version 16.0 includes very significant new core language features to make it easier to implement high performance, functional, elegant array-oriented solutions!