New York Dyalog Meetup

Posted on by

I am very pleased to announce the creation of the New York Dyalog APL Meetup group, details of which can be found online at https://www.meetup.com/New-York-Dyalog-APL-Meetup/. The meetup has been created and is organised by Paul Mansour, who is also sponsoring the venue for the inaugural meetup, scheduled for 6-9pm on Thursday February 7th, at Alley, 119 West 24th Street, New York. If you are interested in meeting APL users in the New York area, please join the Meetup group so that you will be notified of future events. Please sign up for events that you intend to attend so we know you are coming!

Meetup is a service used to organize online groups that host in-person events for people with similar interests, including programming languages. In addition to the New York group, there is also an APL Meetup group in Frankfurt which meets regularly. We welcome the creation of more local meetups! If you create one in your area, remember to inform us at Dyalog so that we can add a link from our event calendar, and arrange to stop by and speak when we are in your neighborhood!

The program for the meetup on February 7th is as follows:

6:00-6:30pm: Time for Networking

6:30-8:00: Morten Kromberg: New Ways of Working with APL

When you are busy solving problems, new technology can be an unwelcome distraction – but every now and again technologies appear which have the potential to make development, maintenance or distribution significantly easier. Morten will demonstrate some of the new ways of working with APL that have become available in the last few years, and also discuss likely features in the next couple of releases of Dyalog APL: 17.1 in Q2 of 2019 and 18.0 in 2020.

8:00-8:15 Short Break

8:15-9:00 Paul Mansour: Keeping it Simple – A Git Workflow for APLers.

Abstract: Git is great, but the newcomer can easily drown in a sea of commands and options. Git doesn’t tell you when or why to branch, when or why to merge or rebase, how to version your project or prepare a release. AcreFlow is a radically simplified Git workflow that answers these questions. It is implemented in Dyalog APL so you can branch, commit, and put out new versions directly from the APL session.

 

Email, RSS Follow

Dyalog ’18 Videos, Week 7

Posted on by

Our stated goal has been to provide variety each week, and I’m happy to say that I have not been able to find any kind of theme in this penultimate set of presentations from Dyalog’18 in Belfast. Although the three technologies presented are all very useful, that is the only thing that they seem to have in common!

FlipDB is a relational database management system that was designed to solve complex data problems from the mortgage and asset finance business, but has grown into a general-purpose toolkit for working with tables. As Paul Mansour writes on the FlipDB web page, mortgages push the limits of many systems due to the quantity of data items, variety of products, constant innovation, and a never-ending stream of file formats and standards. Answering his own question, “What is wrong with SQL?”, Paul shows how easy it is to solve a number of programming puzzles from the Alteryx Weekly Challenge web site using the combination of array and object-orientation available in FlipDB. Just think about how difficult it would have been to do the same with SQL!

Co-dfns is a PhD project at Indiana University, executed by Aaron Hsu and partly funded by Dyalog Ltd. The aim is to build a concurrent compiler for dfns. Aaron regularly presents progress reports at Dyalog user meetings. This year, his focus has shifted to wrapping up demonstrations of completeness and other reporting required to finish his thesis – this will describe his compiler, which is designed to self-host on Data Parallel hardware. Who could have thought that analysing a lexically-scoped, dynamic, language could be so interesting (at least if you want the compiled code to run fast)?

A Jupyter Notebook is a formatted document containing live code, equations, visualizations and narrative text. As Adám Brudzewsky explains in his talk, Jupyter is an open framework that supports “language kernels” for many languages, including Dyalog APL. In other words, a notebook can contain APL expressions and their results – and the expressions can be edited and re-executed by the reader on demand. The expressions can either return textual output which is incorporated into the document, or HTML including SVG graphics – for example, output from SharpPlot or any tool that can produce HTML output.

Summary of this week’s videos:

Email, RSS Follow

Dyalog ’18 Videos, Week 6

Posted on by

Happy New Year – and Welcome to the 6th week of Dyalog ’18 video releases!

If you enjoy geometry, 2019 starts with a couple of real treats; one which builds up to the use of complex numbers just before the end, and another which starts with them and moves on to Quaternions. Alternatively, if you think vectors and matrices containing imaginary numbers are a bit esoteric, what could be more “down to earth” than taking a look at various ways to efficiently extract data from Excel spreadsheets? Finally, we have a talk on a Theory of Everything, which will obviously interest everyone!

Returning to the maths: Nic Delcros asks a seemingly trivial question about the number of dimensions of a vector. As any APLer knows, a vector is a list of numbers and, therefore, has 1 dimension, but of course the numbers in a vector nearly always represent a structure of higher dimensionality. Nic takes us on an entertaining exploration of the case where the numbers represent a dynamic event, where one of the dimensions is time – punctuated with beautiful images.

Dieter Kilsch from the University of Applied Sciences (Technische Hochschule) in Bingen obviously enjoys teaching mathematics! In this talk, he actually managed to make me think that I had some insight into why the Irish mathematician William Hamilton invented the Hamiltonian number system (which is populated by Quaternions), and how it allows us to do algebra on points in a 3-dimensional space, similar to the way complex numbers work for 2 dimensions. For example, Quaternions can be used as a tool of thought and computation for image recognition!

Returning to the very real world, Richard Procter is back with an updated talk on “Excel Mining”, following on from his talk at Dyalog ’15 in Sicily. Like many of us, he frequently needs to load data which originates in Microsoft Excel into APL for processing – and sometimes write back to Excel. Richard has tried a variety of different techniques and provides a list of questions that might decide which technique to use in a given scenario (and performance measurements as well).

It should be no big surprise that John Daintree’s big TOE is not something he needs to take a shoe off to demonstrate. Rather, the Theory Of Everything is a unifying idea that might one day replace a large number of system functions, “root methods” and I-Beams which currently allow programmers to ask questions about the Universe that they are running in. The result will hopefully be a system that is more powerful, but simpler and much more self-documenting than the collection of tools that it would replace.

Summary of this week’s videos:

 

Email, RSS Follow

Dyalog ’18 Videos, Week 5

Posted on by

We are releasing four talks this week from Dyalog ’18 in Belfast. Three of them focus on techniques for deploying APL applications across all the platforms supported by Dyalog APL, in particular Microsoft Windows, Apple macOS, and GNU Linux on x64 platforms (many tools will also work under IBM AIX and on the Raspberry Pi).

If that isn’t your sort of thing, we invite you to relax with Dr. Charles Brenner’s entertaining talk on The Manly Chromosome – How Simplicity can be Confusing. Charles is a “purveyor of forensic mathematics”; he participates in CSI (Crime Scene Investigation) using mathematics. When he actually needs to compute a number for a court case, he often does it in APL (did you know how useful ×.! is?). Although the underlying subject is a serious matter – the mis-interpretation of DNA data related to the Y Chromosome in criminal cases – the talk is full of humour, although you will need to concentrate as some of jokes are rather understated!

Returning to the theme of the week, my own second presentation follows on from my opening keynote. In the keynote, I whizzed through a flurry of scenarios in which a new user of APL was able to run, edit and debug APL code under Linux without actually installing APL, and then deploy the code as a Web Service using JSONServer, and finally as a Web Application using MiServer. The keynote intentionally did not explain much about the technology. In this week’s talk, entitled Cloud Computing, I reveal some of the details of how the smoke was generated and the mirrors were polished for the opening talk.

As mentioned, Dyalog APL is available on a variety of platforms. Our goal is to allow you to freely select a platform for development, and independently select a platform for deployment. For example, you might develop and test the application under Windows or macOS, and deploy it to the cloud under Linux. If your application has a user interface, you may be interested in Brian Becker’s talk on Cross Platform User Interfaces, where he presents a project with the internal code name “DUI” (Dyalog User Interface). We’ve been working on DUI for several years as a component of our Web Server framework MiServer. Brian demonstrates the results of this endeavour: the new HTMLRenderer, which embeds the Google Chrome Browser Engine (known as Chromium) inside Dyalog APL for Windows, macOS and Linux. The same code can now be used for MiServer and HTMLRenderer, allowing it to drive UI without a Web Server using the same principles as Node JS.

If your application is – or can expose – an engine without a user interface, one of the options for deployment is to run it as a Web Service. This makes it callable from anywhere that can connect to it via a network, for example using the JSONServer as demonstrated in my opening keynote. Marko Vranic’s talk on Severless APL demonstrates the latest fashion in deployment technology, where individual functions – rather than applications – can be deployed to the cloud (or a collection of internal servers). Obviously, the name is a little misleading since the functions do run on servers. The point is that you do not need to define a new server or even a container for your application; once the Kubeless environment is set up, developers can continuously deploy new or modified functions into the “serverless” framework from the command line. The framework takes care of encryption of network traffic, allows you to control access to your functions, and lets you manage scaling and load balancing. This talk probably contains a bit more technology and more moving parts than many APL users will be comfortable with, but the important point that Marko makes is that we have all the pieces required to use APL in scenarios where extreme security and scalability are a requirement.

Summary of this week’s videos:

Email, RSS Follow

Dyalog ’18 Videos, Week 4

Posted on by

This week is mostly a deep dive into the new world of storing source code in text files rather than workspaces and other “binary” formats. However, in case that is not your cup of tea yet, we can offer you another talk by Marshall Lochbaum, who presents more amazing algorithms to make the very widely used primitive search functions ∊, ⍳ and ⍸ run faster than ever before. By combining non-branching algorithms with vector instructions and a technique known as Robin Hood Hashing, Marshall is able to drive a modern CPU close to the theoretical maximum throughput, and in many cases spend less than one nanosecond searching for each item of an array.

Source code in text files is the dominant theme, and we are fortunate enough to have three pioneers to show us the way: Paul Mansour, Gilgamesh Athoraya and Kai Jaeger.

Paul has been working on – and using – source code management systems for decades. Recently, his team have implemented a lightweight version of the Acre project management system, named Acre Desktop, based entirely on textual source files. Apart from having to start your day by ]Open-ing a project, rather than by )LOAD-ing a workspace, there are very few changes to how you would actually use APL – but now you have access to a huge collection of professional tools developed for programmers using other programming languages, such as GitHub.

One of the very significant advantages of the APL community starting to use common structures for source code – and projects – is that it becomes realistic to share tools and utilities.
Following on from Paul’s talk, Gilgamesh Athoraya demonstrates a prototype of an APL Package Manager (APM). The APM connects to a repository of packages written in APL and allows you to declare package depenedencies from a public or private repository. It also keeps tabs on the availability of new versions of dependencies, and allows you to easily update them when the time is right.

A package manager can only be successful if there are packages to be managed. Kai Jaeger has been an APL Toolsmith for a very long time, and made much of his work available via the APLTree. Now, Kai has transferred the contents of the APLTree to GitHub, making everything available as textual source. With a bit of luck, once the APM finds its legs, we’ll all be able to use Acre Desktop to define projects, Git[Hub/Lab] to manage the source, and APM to search for Kai’s tools and manage our dependencies on them!

Summary of this week’s videos:

Email, RSS Follow

Dyalog ’18 Videos, Week 3

Posted on by

The four presentations from Dyalog’18 that we are releasing this week address both the visible (user interface) and invisible (performance) parts of application design. Starting with performance:

“You don’t have to be an engineer to be a racing driver, but you do have to have Mechanical Sympathy.” – former Formula One racing driver Sir John Young “Jackie” Stewart, OBE


This quote was at the heart of the talk by our invited keynote speaker Martin Thompson. In order to write software which performs well, you need to have a basic understanding of how the underlying machinery works. Understanding basic mathematical models for the theoretical throughput of software and hardware helps us take the step from being alchemists to scientists, as we endeavour to write high-performance systems.

Martin takes us for an entertaining stroll through the evolution of modern processors, and some of the maths behind high performance systems. The good news is that systems which make sequential and predictable memory accesses are likely to find sympathy with modern hardware…

Marshall Lochbaum, the most recent addition to the core interpreter team at Dyalog, followed up with a talk on a number of his ideas for increasing the mechanical sympathy of Dyalog APL, to take maximum advantage of branch prediction and other features of modern processors. Some strategies take advantage of runtime inspection of the arguments, something that is more natural in an interpreter with the ability to dynamically select data types, as opposed to strongly typed strategies typically employed by compilers.


TamStat is an application which helps students Tame Statistics. In two talks at Dyalog’18, Stephen Mansour and Michael Baas focus on two different aspects of the user experience. In the first talk, Stephen focuses on the notation available to users of TamStat. Where many statistical libraries contain dozens of strangely named functions with a variety of switches and parameters, TamStat uses a small set of functions, combined with another small set of operators, to provide a very simple but extremely elegant notation for computing probabilities based on a wide variety of distributions. For example:

⍝ Probability that 7 coin flips (0.5 specifying a "fair" coin) will result 
⍝ in at least 3 heads:
7 0.5 binomial probability ≥ 3

⍝ Probability that a number from a normal distribution with a mean of 0 and 
⍝ standard deviation of 1 will be ≤ 3:
0 1   normal   probability ≤ 3

I almost wish I could go back to University and start Statistics 101 again 😊.


Notation is a powerful tool of thought, but graphs make it easier to visualise the results. Following Stephen’s talk, Michael Baas describes work that Dyalog is doing in collaboration with Stephen, with the goal of wrapping TamStat in a modern, HTML/JavaScript based frontend. Current TamStat is based on the ⎕WC (Window Create) library function and is therefore restricted to running on Microsoft Windows. However, many of Stephen’s students use Mac or Linux laptops. The new interface also makes it possible to run TamStat as a web-based service with a web site. We expect that this work will make TamStat accessible to a much wider audience.

Summary of this week’s videos:

Email, RSS Follow