Announcing Dyalog Version 12.1

Version 12.1 will be released in the 2nd half of 2009 – if all goes well it will be out before the Dyalog '09 conference starts on September 13th. We have not quite finished development of this version, but are now able to summarize the most important features of Version 12.1:

Improved Editing and Debugging

The most visible change to v12.1 will be enhancements to the editor used for editing functions, namespace and class scripts.

• The editor now supports outlining, with collapsible regions, allowing easier editing of large functions: Control structures can be collapsed into a single line, and user-defined regions make it possible to hide sections of code which are irrelevant in a particular context.
• We've added support for selecting a group of lines in a function and turning them into a sub-function, allowing you to decide which variables referenced in the line should become results, arguments, or local/global variables.
• A keystroke skips from one end of a region (like a control structure) to the other.

 

 

We have added a number of new features aimed at making it easier to edit (large) class and namespace scripts – resolving a number of usability issues identified by "early adopters" of the object-oriented features introduced in version 11.0.

• An explorer-style tree allows fast navigation within the elements contained within a class or namespace.
• Within a class or namespace script, each function is a collapsible region.
• The command )ed Class.Fn now edits a function (which could be a new function) within a class, rather than creating a new function in the namespace surrounding the class.
• Classes can be fixed even when they contain errors – and a new menu item allows fixing of a class or namespace without executing initialization code.
• The speed of syntax colouring has been significantly increased and now copes without problems with scripts having tens of thousands of lines.

Most of the new editor features are available both in the GUI and TTY-based session managers.

Faster with One Core – Faster Still with Several

A number of important primitive functions have been overhauled to take advantage of new algorithms and features of modern processors, and have become significantly faster. Most scalar dyadic functions will use all available cores for floating-point operations, which will speed up many arithmetic operations on large arrays.

The following language elements have been revisited in Version 12.1:

• Indexing and Indexed Assignment consume very much less memory than before, and have been speeded up in a number of cases
• Many Boolean Reductions and Scans have been very significantly speeded up (see Roger Hui's Dyalog '08 presentation)
• Transpose and Compress are 2-3 times faster in many cases
• Grade up / down, and four "sort" idioms: {⍵[⍋⍵]},{⍵[⍒⍵]},{⍵[⍋⍵;]},{⍵[⍒⍵;]}
• Many cases of addition, subtraction and multiplication
• Matrix multiplication (+.Χ)
• Set functions (∊⍳~∩∪), where one of the arguments has 3 items or less (no hashing)
• Depending on array sizes and types, most scalar dyadic functions will use multiple cores when available: | * ⍟ ! χ ≠ ⌈ ⌊ < ≤ = ≥ > ∨ (gcd) ∧ (lcm)

As an example of a primitive function which has been speeded up, the following graph compares the performance of grade up and sorting on integers in v12.1:

 

 

Enhanced File Journaling and Crash Recovery

Version 12.0 added file journaling features which effectively protect component files against damage due to APL being terminated unexpectedly. Version 12.1 adds checksums, and additional levels of journaling which are designed to make component files robust even in the face of catastrophic failures of the operating system, in which in-memory caches are not committed to disk.

Version 12.1 makes it possible to add a checksum to each component. The new system function ⎕FCHK is able to identify and recover all undamaged components from files which contain checksums. ⎕FCHK replaces the qfsck auxiliary processor, and when combined with checksums, ⎕FCHK is able to reliably detect all – and repair most – damage.

Even without journaling enabled, ⎕FCHK will be able to repair most file damage caused by APL terminating unexpectedly if a file contains checksums. At level 1 (the only level available in version 12.0), no recovery is required unless the operating system crashed or the machine suffered a catastrophic hardware failure which prevented cache from being flushed to disk.

Levels 2 and 3, which are new to Version 12.1, call the operating system function fsync (once per file operation for level 2, twice for level 3). As a result, these levels of journaling slow the file system down significantly – roughly as much as the common technique of explicitly flushing the cache or closing and re-opening files in order to ensure that writes are forced to disk. Unlike the latter techniques, which simply reduce the risk of damage, it is intended that level 2 guarantees that no data is lost if ⎕FCHK is used to perform "crash recovery" after an operating system failure. Level 3 is designed to guarantee that – even without crash recovery – no data will be lost unless the disk system itself has failed.

Using the file system function ⎕FPROPS, journaling options can be set differently for each component file – and changed dynamically. This allows you to balance security and performance depending on the requirements of your application and the type of data in each file.

Fast XML Conversion

A new system function ⎕XML will provide efficient conversion of XML strings to and from APL arrays, to support web applications and other components that need to exchange data in XML format. The ⎕XML function has been designed in close collaboration with the APLX team at MicroAPL, and in consultation with other colleagues, and we expect more or less identical functionality to appear in APLX – and possibly other APL systems in the future.

Use Microsoft.NET Controls in your Dyalog GUI Forms

Version 12.1 will allow you to tap in to the vast array of Microsoft.NET-based GUI components provided by Microsoft and numerous "third parties" – without rewriting your entire application to use the Microsoft.NET framework: With v12.1, it will be possible to embed Microsoft.NET GUI controls within existing Dyalog GUI forms.

Save APL Applications as a Single .EXE File

Distribution of many Dyalog-based applications will become significantly easier with Version 12.1, which will have an option to create a single file which contains an APL interpreter and an APL workspace bundled together.

IPv6 Support with Conga v2.0

A new version of the TCP connectivity tool known as Conga will be included with version 12.1: Conga v2.0 will provide support for IPv6, which is a new standard for IP addresses. IPv6 is being introduced because the current 4-byte (nnn.nnn.nnn.nnn) address format is rapidly running out of free addresses. The new Conga also sports improved thread scheduling when handling large numbers of connections, and an improved API which will allow parameters to API functions to be named.

User Command Processor

Integrating “third party” development tools with your Dyalog environment will become even easier in Version 12.1, which will introduce support for “User Commands” entered at the six-space prompt, when preceded by a closing square bracket:

      ]load study/files/KeyedFile

Spice will continue to be the default framework for adding development tools, and all Spice commands will be available via the new mechanism, which will replace the separate graphical "Spice Command Line" used by Spice in Versions 11 and 12. This means that Spice can easily be used in development environments with no GUI support.

Web Service Implementation

Web Services are components which use open standards XML and SOAP to describe and provide components written in any language via the internet. These standards are supported by most common development tools. Thus, if you expose selected functions in your APL application as a Web Service, they can easily be called from almost any other application – even if that application is running on a different machine, and that machine has a completely different architecture.

Version 12.1 will include a workspace which is based on Conga and the new ⎕XML system function, and provides an efficient implementation of both the client and server side of the Web Service protocol. The client allows you to call other web services, and the server makes it straightforward to expose your application as a Web Service. The workspace is called a stand-alone Web Service because it allows APL to offer web services without requiring a web server like Apache or Microsoft IIS as a front end (you can, of course, decide to put your service "behind" a web server if you wish). The SAWS workspace supports secure web services using the TLS/SSL security layer provided by Conga.

An important part of running a Web Service is to have the operating system start it automatically on a system restart. The Web Service implementation will include a tool that allows it – or any other APL application – to be registered as a Windows Service, so that the application can be started automatically when Windows starts, and be managed from the system console.