You can create a sort of "model of a model railroad" with the program, and it serves no other purpose than having fun.

The sources can be found on GitLab or downloaded from https://software-lab.de/bahn.tgz (source code within the "misc/"-folder). PicoLisp version >= 23.12 is required.

Concept

The base of the simulation is a railroad network drawn in ASCII, with each line representing a track. The cyan lines represent switches that enable the trains to change tracks.

The trains are the simulated objects ("bots"). They are depicted as @000 , where the @ mark represents the locomotive and each 0 represents a wagon.

At any point in time, a train is either

• waiting, e.g. at a station,

• moving from one position to another,

• or shunting (rearranging locomotive and wagons).

Let's see it in action in the example below. There are 5 trains in operation - some with multiple wagons, some with a locomotive at each end, and one solitary locomotive without any wagons.

For each train an individual schedule has been defined based on a specified order of positions. However, the precise movements of the trains are determined dynamically using Discrete Event Simulation.

The GIF below is showing the first 30 seconds of the simulation running at 8x speed.

Running the simulation

The source code is split into two files:

• bahn.l, containing the fundamental logic for the train and network simulation.

• plan.l, containing the railway network layout and driving parameters, such as the number and specifications of trains, schedule, speed, etc.

Note: if you are confused about the wording - "bahn" is the German word for train :)

To start the simulation, use the following command:

$ pil bahn.l -bahn~main plan.l -go +

Typing "s" starts and stops the simulation, while the simulation speed doubles and halves with "+" and "-" respectively. ESC will drop you into the REPL, and calling (go) from the REPL will continue with the simulation. To exit the simulation, press ESC and then Ctrl-D or (bye).

Train Simulation in Detail

A train can be in three states: Waiting, moving and shunting.

Waiting is realized using the pause function of the DES library.

More interesting is moving, which consists of several steps:

1. Acceleration until reaching travel speed.

2. Continuous travel at the designated speed.

3. Deceleration upon reaching the destination.

Discrete Event Simulation offers several advantages over continuous simulation, particularly when dealing with numerous bots with varying timings. For instance, continuous simulation will use a fixed time slice dt and the bot's current speed v to calculate their change in position, using the following differential equation:

This calculation is performed for all bots, regardless of their individual speeds.

In contrast, discrete event simulation utilizes a spatial resolution ds (which may vary for each bot and scenario) and computes the time required to "pause" in the simulation until the bot has covered the distance:

Now, all moving bots remain inactive for their scheduled amounts of time - longer if they move at slower speeds and/or coarser resolutions. This approach can significantly reduce processing time.

Note that the equations above are valid only for moving at constant speed. For accelerated movement, a continuous simulation would calculate:

For discrete event simulation, we can use:

This behavior is implemented in the drive function in the file bahn.l. The function takes the following parameters: acceleration (in m/s², also utilized for deceleration during braking), travel speed (in m/s), and X and Y coordinates for the destination. Optionally, additional X and Y destination pairs can be provided to enforce intermediate stops.

For each destination, the function searches for a viable path through the track network and attempts to lock the path. If the path is unavailable, it enters a pause state until receiving a signal that another bot has released the locks. It also toggles all necessary switches as required for its route.

Now the last train state, shunting, which is implemented in the shunt function in bahn.l. This function disconnects the locomotive from the first wagon, calls drive and turn repeatedly to navigate to the other end of the train, and connects to the previously last wagon.

As an example, this is how the first locomotive is defined in bahn.l.

(new '(+Locomotive) (pd 1 10 'a) 5
   (pause)
   (loop
      (drive 0.2 40.0  31 14)
      (pause 12.0)
      (drive 0.2 40.0  45 34)
      (shunt
         (drive 0.1 10.0  45 38)
         (turn)
         (drive 0.1 10.0  48 32  45 23)
         (turn)
         (drive 0.1 10.0  45 28) )
      (drive 0.2 40.0  31 19)
      ...  ) ) )

The pd method of the locomotive object defines that the locomotive starts at position (x, y) = (1, 10) and faces in direction of a, e. g. downwards. The last parameter of the first line, 5, indicates the number of wagons.

As mentioned above, the drive method takes four parameters: acceleration, speed, and a target x-y-position. drive is also used within the shunt method to define the movement of the locomotive while the wagons stay at their current position.

With this, we have covered all possible states of the train. Next we will see how the track network is implemented.

Track Simulation in Detail

For the railway tracks to run the trains on, we use the track network library available in the PicoLisp distribution. The library has a tracks function in @lib/simul.l, which processes the ASCII representation of a railway layout from the current input stream and generates an appropriate data structure.

The idea is to have interconnected track elements where the bots can be positioned, along with a cdr-like operation to move a bot from one element to the next. Iterating this operation results in a continuous movement into one direction.

However, for a comprehensive track network, a simple linked list of track elements is insufficient. It must be possible to traverse it in both directions and accommodate switches (turnouts) and loops. Loops present a particular challenge. Consider this layout where a bot starts in the top-right position and moves leftwards:

             <--               <--
       ---------------------------------
      /                 /      -->
     /                 /
    /                 /
   |                 |
   |                 |
   |                 |
    \               /
     \             /
      \           /
       -----------
           -->

After traversing the loop, it ends up in the top-right position again, but now moving in the opposite direction! The cdr operation would throw it back into the loop instead of continuing in the intended direction.

To address this, the tracks function generates what we call a "Networked Linked Lists" data structure. This structure is a form of indirect doubly linked list that accommodates branches, allowing for traversal in both directions and supporting complex track layouts such as loops.

The structure can be visualized with the built-in ASCII diagramming capabilities of the PicoLisp Vip editor. In the PicoLisp REPL (in debug mode!) enter

: (vi "@doc/Tracks")

and type "v" to view it in a new buffer. It will show:

Connectors:

---------+                 +------------->   +----------------------------->
         |                 |                 |
         v                 |                 |
      +-----+-----+     +--+--+-----+     +--+--+-----+
  +-->|  |  |  ---+---->|  |  |  ---+---->|  |  |  |  |
  |   +--+--+-----+     +-----+-----+     +-----+--+--+
  |      | ^                                       |
  |      | |                                       |
  |      | +---------------------------------------|-----------------+
  |      |                                         |                 |
  |      |                                         |                 |
  |      |      +----------+ +---------------------+                 |
  |      |      |          | |                                       |
  |      v      v          | v                                       |
  |   +------------+    +--+--+-----+     +-----+-----+     +-----+--+--+
  +---+- a      b -+--->|  |  |  ---+---->|  |  |  ---+---->|  |  |  |  |
      |            |    +-----+-----+     +--+--+-----+     +--+--+-----+
      |   Track    |       ^                 |                 |
      |            |       |                 |                 |
      | x        y |       +-----------------|-----------------|------------
      +------------+                         |                 |
                                             |                 |
<--------------------------------------------+   <-------------+

As you can see, it is a rather complicated structure. The Track box is a symbol holding the actual track element. It stores information like the x and y coordinates and connectors to the neighboring track elements in a and b. Each connector consists of three cells for forward- and backward-links and optional switches in both directions.

User Interface

Showing the animated ASCII drawing and handling user keyboard input is all done by a single function called display. It is called by go in a loop until ESC is typed, and allows to move around in the layout or along a rail track, and to toggle switches manually. Additionally, the following commands can be used in order to interact with the simulation:

• s stops or continues the simulation.

• + and - double and halve the simulation speed respectively. Initially, the speed factor is 2. It is displayed at the bottom left.

• A red `#` cursor (initially hidden on the left outside the layout) can be moved around on the screen with the following keystrokes:

  • j or DOWN, k or UP, l and h move by a single position

  • z and Z move by eight positions right and left

  • w and b move by a "word" (e.g. rail element) to the right and left

  • 0 and $ to go to left and right end of the layout

  • g and G go to top and bottom of the layout

  • TAB and BACK to jump to the next and previous switch

  • ENTER to toggle a switch

  • SPACE to move along the track

  • r to toggle the move direction for the SPACE command.

Now, with this information you should have a good understanding of how to create a railway model using discrete event simulation in PicoLisp. By referring to the examples on GitLab, tweaking it into your own railway model shouldn't be too challenging.

With some (considerable) effort, you could even replace the ASCII representation by something more visually appealing 😊

Sources

• https://www.irasutoya.com/2013/05/blog-post_9988.html

• https://software-lab.de/doc/des.html

• https://pixabay.com/de/photos/modelleisenbahn-modell-eisenbahn-1146828/

Discrete Event Simulation in PicoLisp

As the name suggests, Discrete Event Simulation (DES) models systems by focusing on a sequence of distinct events. Instead of continuously tracking a system's state, DES jumps from one distinct event to another. This approach is frequently employed in gaming to simulate aspects like character behavior and crowd movements. By focusing on these key events, it gets much easier to understand and design complex behaviors without tracking every little detail.

Since pil21 version 22.1.18 (January 2022), PicoLisp has a DES library. You can find the code in lib/simul.l and the documentation in doc/des.html.

Using the DES library, we can simulate the behavior of agents which are realized as coroutines. These coroutines are driven by events. To manage these coroutines, we have several control options:

• pause: pauses the coroutine either for a set duration or until a designated event takes place.

• event: Sends a signal to all coroutines awaiting that specific event.

• wake: Resumes a previously suspended coroutine.

• des: enables the simulation to "jump" from one event to another.

To understand how DES works, let's look at a famous programming problem: E. W. Dijkstra's "Dining Philosophers".

Note: The link to the final code can be found at the bottom of the page.

The Dining Philosophers

Task Description

Five silent philosophers sit at a round table with bowls of spaghetti. Forks are placed between each pair of adjacent philosophers. Each philosopher thinks deeply and, intermittently, and interrupts their thoughts to eat from a bowl of spaghetti placed in front of them. However, to eat, they need two forks. A single fork is placed between each pair of adjacent philosophers. The crux of the problem is that each philosopher can only eat when they have both their left and right forks.

The problem was designed to illustrate the challenges of ensuring that several concurrent processes (e.g. the philosophers) can access shared resources (the forks) without conflict:

1. No two philosophers should be able to pick up the same fork at the same time (preventing race conditions).

2. Philosophers should not starve (fairness in accessing resources).

3. Deadlocks should be prevented (e.g. all philosophers having one fork in their left hand).

The Dining Philosophers is one of my favorite problem descriptions because it leaves so many questions hanging in the air: Why do they use two forks? Do the forks get cleaned in between? And which type of spaghetti is being served?

(Unfortunately, these questions will not be answered by the algorithm).

Because this is a modern blog, we will host a gender-parity party. Our dinner guests are Hannah Arendt, Hildegard v. Bingen, Rosa Luxemburg, Immanuel Kant, Friedrich Nietzsche and Ludwig Wittgenstein.

Loading the simul.l library

To use the DES functions, we need to load the library and declare the namespaces:

(load "@lib/simul.l")
(symbols 'dining-philosophers 'simul 'pico)

The symbols function defines the search hierarchy for symbols (e.g. variables, functions...). Initially, it searches within the local namespace, labeled dining-philosophers. If not found, it proceeds to the simul namespace specified in simul.l, and finally checks the pil21 namespace pico.

Defining the agents

Simply put, in our simulation each philosopher acts as an agent. And as previously mentioned, every agent operates as a coroutine. These agents have a singular task - dining - and to do so, they require two forks.

The forks are shared within all agents, e.g. they are globals. Following the PicoLisp conventions, we declare these global variables at the beginning and their name starts with an asterisk.

(local) (*ForkA *ForkB *ForkC *ForkD *ForkE *ForkF)

(co 'Arendt
   (dining '*ForkA '*ForkB) )
(co 'v.-Bingen
   (dining '*ForkB '*ForkC) )
(co 'Luxemburg
   (dining '*ForkC '*ForkD) )
(co 'Kant
   (dining '*ForkD '*ForkE) )
(co 'Nietzsche
   (dining '*ForkE '*ForkF) )
(co 'Wittgenstein
   (dining '*ForkF '*ForkA) )

(Since it's a round table, Wittgenstein is sharing his fork with Hannah Arendt.)

The "dining" function

Now to the heart of our program. We can define five stages for each agent:

1. Thinking: The philosopher is in a contemplative state and isn't looking to eat.

2. Hungry: The philosopher feels the need to eat.

3. Waiting for Forks: The philosopher waits to acquire the two forks if they're unavailable.

4. Picking Up Forks and Eating: Once the forks are free, the philosopher picks them up and begins to eat.

5. Putting Down the Forks: After eating, the philosopher releases the forks for others to use.

These five stages are continuously looped throughout the simulation.

The initial stage is straightforward: We display "<philosopher> is hungry..." and then pause the simulation. Assuming our simulation time is measured in minutes, each pause duration ranges between 180 and 240 minutes.

(de dining (LeftFork RightFork)
  (loop
    (prinl (pack (co) " is thinking... 💭"))
    (pause (rand 180 240))

Note: To extract the coroutine tag we use (co), and the pack function converts all arguments to a single string.

Now we can simulate 20 event steps by calling the des function 20 times.

<...>
(co 'Wittgenstein
   (dining '*ForkE '*ForkA) ) 

(do 20 (des))

Starting the script, we find all philosophers in deep thought.

$ ~/pil21/pil dining-philosophers.l
Arendt is thinking... 💭
v.-Bingen is thinking... 💭
Luxemburg is thinking... 💭
Kant is thinking... 💭
Nietzsche is thinking... 💭
<...>

After between 180 to 240 time units, the coroutine resumes. Now is the time to declare the next step: The philosopher is getting hungry. If both forks are available, they proceed to eat.

For our simulation, the act of "eating" involves setting the global fork values to a non-NIL value. It is convenient to set the value to the coroutine's tag, e.g. the philosopher's name:

(set LeftFork (set RightFork (co)))
(prinl (pack (co) " is eating! 🍝"))
(pause 20)

First, we need to check if both forks are free. If one or both forks are taken, our coroutine pauses. It will start again when it gets an event signal related to LeftFork or RightFork.

(while (or (val LeftFork) (val RightFork))
   (prinl (pack (co) " has no fork!! 🤬🤬🤬" ) )
   (pause LeftFork RightFork) )

Note: This part will be changed later to something more expressive.

Fortunately for our hungry philosophers, each meal lasts only 20 minutes. After that, the philosopher releases the forks. This action sends an event signal for the paused coroutines to resume:

(prinl (pack (co) " finished eating.")) 
(set LeftFork (set RightFork NIL))
(event LeftFork)
(event RightFork)

This is all we need for a fully functional simulation:

$ ~/pil21/pil dining-philosophers.l
Arendt is thinking... 💭
v.-Bingen is thinking... 💭
<...>
Nietzsche has no fork!! 🤬🤬🤬
Arendt finished eating.
Arendt is thinking... 💭

This simulation avoids deadlocks by two factors: both forks are always handled simultaneously, and coroutines waiting for forks are consistently queued behind others that are already waiting.

Managing the time

We've outlined the basic program structure but have yet to address an important aspect: Time simulation.

Within DES, there are two modes: Realtime and Simulated Time. These modes are determined by the *Rt global variable. If *Rt is set to NIL, it indicates simulated time. A value of 1 represents real-world clock speed, 2 double speed, and so on.

Since we do not want to wait 4 hours for all philosophers to finish thinking, let's stick with simulated time. To do so, we set *Rt to NIL at the beginning of our program:

(off *Rt)

To provide better insights into our program's process, we want to include a timestamp in our output. For this we introduce a function called now that prints the current time in HH:MM format, followed by the coroutine tag and a specified string. We can utilize the built-in tim$ function for this purpose.

(de now (Str)
   (prinl (tim$ (* 60 *Time)) " " (co) " " Str) )

If we replace prinl with the now function in our code and run the program, the output will show timestamps. This makes it easy to verify that the thinking phase indeed lasts at least 3 "hours" and the eating phase takes exactly 20 "minutes".

$ ~/pil21/pil dining-philosophers.l                                                                                                                                          
<...>
03:18 Nietzsche is hungry... 😋
03:18 Nietzsche has no fork!! 🤬🤬🤬
03:20 Arendt finished eating.
03:20 Arendt is thinking... 💭
03:20 Wittgenstein has no fork!! 🤬🤬🤬

As a last little gimmick, let's also add which fork is missing and who is using it:

 <...>   
    (use (L R)
      (while
        (or
          (setq L (val LeftFork))
          (setq R (val RightFork)) )
        (now (pack "waiting for " L (and L R " and ") R " 🤬🤬🤬 "))
      (pause LeftFork RightFork) ) )

Which adds quite some drama to the dinner table!

11:05 Luxemburg is hungry... 😋
11:05 Luxemburg waiting for Kant 🤬🤬🤬 
11:21 Nietzsche is hungry... 😋
11:21 Nietzsche waiting for Kant 🤬🤬🤬

Adding the main function

Having a main function is not a must, but it makes debugging the program easier. To set it up, we start by defining the main symbol. Next, we set up the namespace search order, and then we add the rest of our code.

(local) (main)

(de main()
  (symbols '(dining-philosophers simul pico))
  (co 'Arendt 
     <...>

Now we can start the script like this:

$ ~/pil21/pil dining-philosophers.l -dining-philosophers~main +  

00:00 Arendt is thinking... 💭                                                                                                                                      
00:00 v.-Bingen is thinking... 💭                                                                                                                                   
00:00 Luxemburg is thinking... 💭                                                                                                                                   
00:00 Kant is thinking... 💭                                                                                                                                        
00:00 Nietzsche is thinking... 💭                                                                                                                                   
00:00 Wittgenstein is thinking... 💭                                                                                                                                
dining-philosophers:

This will directly open a PicoLisp REPL in the correct namespace for debugging. Now you can verify the stack or run some more simulation steps with (des).

You can find the final code here.

Sources

• https://software-lab.de/doc/des.html

• https://www.irasutoya.com/

• https://software-lab.de/doc/index.html

What are Coroutines?

Coroutines are a type of programming construct that aim to achieve concurrency in programming without using threads or processes. The key idea of coroutines is that their execution can be paused and resumed at specific points during their execution, allowing other coroutines to run in the meantime. This requires that the local environment as well as the state of control of the function are preserved while the coroutines are stopped.

One of the benefits of coroutines is that they can be used to write asynchronous code in a more sequential and readable manner, compared to other concurrency models such as threads or callbacks. Coroutines can be used to write code that looks like synchronous code, while still taking advantage of asynchronous behavior and avoiding some of the pitfalls of concurrent programming, such as race conditions and deadlocks.

Co-Routines in PicoLisp

We need two functions to create a coroutine in PicoLisp: co to initiate the coroutine, and yield, which controls when the function is stopped and resumed.

From the documentation:

(co ['any [. prg]]) -> any

Starts, resumes or stops a coroutine with the tag given by any. (...) prg will be executed until it either terminates normally or until yield is called.

Let's look at a very simple co-routine:

(co 'a
    (let N 0
        (loop
            (yield
                (inc 'N) ) ) ) )

According to the definition, we have two parts: a is the tag of the coroutine, and (let N 0 (loop (yield (inc 'N))))) is its program body. If we call it for the first time, N will be set to zero and then increased by one in a loop. Then yield interrupts the coroutine and returns the value.

Let's test it in the REPL:

: (co 'a (let N 0 (loop (yield (inc 'N))))) 
-> 1

As expected, the return value is 1, because N has been increased once.

Now comes the point: Next time we call the coroutine a, it will resume where we stopped. The syntax to resume is (co <tag> <anything>) , for example:

: (co 'a T) 
-> 2
: (co 'a T)
-> 3

Note: Theanythingpart can be literally anything as it does not get executed. For example, this one will also work but does not make sense:

: (co 'a (println "this will not print anything"))
-> 4

To stop the coroutine, we call (co <tag>) without any argument. If then the coroutine gets called again, it will start from the beginning.

: (co 'a) 
-> a
: (co 'a (let N 0 (loop (yield (inc 'N))))) 
-> 1

Managing the Stack

To keep the local environment for each coroutine, each one of them needs its own stack space to maintain its state. This means that for large and nested coroutines, it might be necessary to monitor the stack size. By default, the stack segment size is 64 kB for coroutines and 256 kB for the main stack segment.

The current stack can be viewed like this:

: (stack)
-> ((a . 63) (T . 250) . 64)

In the output above we see our coroutine a and the main routine (which also maintains the REPL) T, with the respective unused stack space, plus the general coroutine stack size:

• coroutine a has unused stack space of 63 kB (out of 64),

• the main routine T has unused stack space of 250 kB (out of 256 kB),

• The allocated stack size of each coroutine is 64 kB.

In some situations, you might want to adjust the stack space. This is done by passing arguments to the stack function, where the first argument is the coroutine stack size and the second (optional) argument is the main stack size. The coroutine stack size can only be modified if no coroutine is running.

: (co 'a)   # stop current coroutine
-> a
: (stack 128 512)
-> 128

If we then start the coroutine again, we will see the updated stack sizes:

: (co 'a (let N 0 (loop (yield (inc 'N)))))   # restart coroutine
-> 1
: (stack)
-> ((a . 127) (T . 507) . 128)

Obviously we cannot allocate more stack space to a coroutine than is allocated by the system to the picolisp process. The default stack space settings can be checked with the ulimit -s command in Linux systems:

$ ulimit -s
8192

In the example above, it is 8192 kB (8 MB). If we try to allocate more than 8 MB to the coroutines, we will get a segmentation fault error:

: (stack 4200 4200)
-> 4200
: (co 'a T)
[1]    17524 segmentation fault (core dumped)  ~/pil21/pil +

To prevent this, we can increase the stack space for the PicoLisp process with ulimit -s <stack space>:

$ ulimit -s 16384   # 16 MB
$ ulimit -s unlimited

Use with care, as allocating too many resources to a process might exhaust your system's resource limits and lead to unexpected behavior.

If you find this "increase a number" example a little bit boring, wait for the next posts - we will see some more useful examples on how to use coroutines soon.

Sources

• https://picolisp.com/wiki/?Coroutines

• http://software-lab.de/doc/ref.html#coroutines

So how to transform the database output into this colorful graph? Luckily, we don't need to re-invent the wheel - data visualization can be very easy for example with a little help from the JavaScript D3.js library.

In the following post, I will explain how I created the little demo page that you can also visit with this link.

About D3.js

D3.js is an open-source JavaScript library that makes it possible to create pretty and flexible graphs in quite an easy way. It is not the only data visualization library - alternatives are for example charts.js or vis.js. But at the point of writing this post, D3.js the project with the largest community and it comes with a pretty good documentation.

With help of D3.js, we can keep the data separate from the visualization. For example, we can retrieve data dynamically from the PicoLisp database or from user input, and feed this data into the JavaScript function. All we need is to define the DOM object where the graph should be created.

A minimal example

Let's take a look at a truly minimal example (no PicoLisp). Let's assume we have a (JavaScript array) var data = [90, 100, 30, 80]. Then the corresponding bar plot will be defined like this:

<!DOCTYPE html>
<html lang="en">
<head>
   ...
<script src="https://d3js.org/d3.v3.min.js" charset="utf-8"></script>
<style> ... </style>
</head>

<body>
<script type="text/javascript">
var data = [150, 230, 180, 90];
var svg = d3.select("body")
   .append("svg")
   .attr("width", 300)
   .attr("height", 200);
svg.selectAll(".bar")
   .data(data)
   .enter()
   .append("rect")
   .attr({
      class : "bar",
      width : function(d) {return d;},
      height: "40",
      y : function(d, i) {return i*50 + 10;},
      x : "10"
   });
</script>
</body>
</html>

And this is how it looks like:

First of all, we need to include the D3.js source file in the header of our HTML page.

<script src="https://d3js.org/d3.v7.min.js"></script>

The actual graph is defined in the second <script> </script> tag. There we create an SVG element with in the specified DOM object (in this case simply<body>). Then we append rect (rectangular) elements to it. The position and size of the rectangles is defined by the values of the data array.

Since this is not supposed to be a JavaScript tutorial, we will not go into further depth with this topic. But there are plenty of great resources about D3.js - you only need to be careful to use the correct version, because the releases are partially not backwards-compatible. This post uses the most recent version D3.v7.min.js.

Calling D3.js from PicoLisp

Now let's recreate the same thing with PicoLisp. First we need to wrap the JavaScript code from above into a function and save it in an external file. As a reference, I created three graphs (click on the link to see the source code):

• Bar graph example
• Pie graph example
• XY-graph example

All three functions take four arguments: the data array, the DOM-id of the div where the graph should be created, and the width and height of the graph.

function drawXYGraph(data, id, w, h) {

      var svg = d3.select(id)
      .append("svg")
      .attr("width", w)
      .attr("height", h);

...

Then we create the PicoLisp main file and load these javascript functions, as well as the D3 library itself.

(load "@lib/http.l" "@lib/xhtml.l" "@lib/form.l")
(setq *Css '("@lib.css" "css/bootstrap.css" "css/custom.css"))

(de work ()
   (and (app) *Port% (redirect (baseHRef) *SesId *Url))
   (action
      (html 0 "D3.JS in PicoLisp" *Css NIL
         (javascript "https://d3js.org/d3.v7.min.js")
         (javascript "js/barGraph.js")
         (javascript "js/xyGraph.js")
         (javascript "js/pieGraph.js") ) ) )

(de go ()
   (server 8080 "!work") )

Note: go back to the previous tutorials in the Web app series if you don't understand this.

You can start the server with:

pil <filename> -go

and should be able to see a blank HTML page on http://localhost:8080. With Ctrl+U (Firefox), you can view the page's source code and double-check that the JavaScript sources are included:

Now, all we need to do is create the data array and call the Javascript function. First of all, we define a global variable *Data (eventually, this data could for example come from a database):

(setq *Data (90 230 180 300 110 195))

Then we create a div with the id xyGraph within the HTML-body:

(<div> '(id . "xyGraph"))

Then we call the JavaScript function we just included. Since JavaScript uses arrays, not list, we need to convert our list to a JavaScript array with help from the glue function. From the docs:

(glue 'any 'lst) -> sym
Builds a new transient symbol (string) by packing the any argument between the individual elements of lst.

    : (glue "," '(a b c d))
    -> "a,b,c,d"

Applied to our example:

(<javascript>
   " var data = [" (glue ", " *Data) "]; "
   "drawXYGraph(data, xyGraph, 400, 400); " )

And we see the graph appearing:

The finished demo page

I have expanded the example above by an input field that allows the user to add new data points to the list, which causes the graph to update automatically. You can test it out here.

The full source code can be viewed here.

Sources

• http://d3js.org
• https://www.tutorialsteacher.com/d3js/create-pie-chart-using-d3js
• https://bl.ocks.org/farazshuja/afd5e5f492fd00dd0ac4644d53716f4e
• https://www.tutorialsteacher.com/d3js/create-bar-chart-using-d3js
• https://software-lab.de/doc/index.html

• https://picolisp-explored.com/an-openstreetmap-app-written-in-picolisp-pt-1
• https://picolisp-explored.com/an-openstreetmap-app-written-in-picolisp-pt-2

The app gets the location data from the GPS module and the ice cream locations via the OpenStreetMap API. This means that there are a lot of external factors which can cause our app to slow down or even stop working. Now, in order to keep the user happy, let's add at least some small feedback which helps the user to understand what is currently going on.

In order to trigger this feedback we will use Server-Sent Events.

What are Server-Sent Events?

In a typical website, the client (for example the front-end of our app) requests the server for content and receives data as response. However, there might be cases when the server needs to push updated data to the client without the client asking for it. For this purpose, server-sent events have been standardized as part of HTML5 by the W3C in 2008. It is supported by all major browsers.

Basically, server-sent events establish a uni-directional connection from the server to the client. It allows to update a specific DOM component without having to reload the whole page. In PicoLisp, we can implement Server-Sent Events with just a couple of lines of code.

Specifying the DOM component in the front end

In the first step, let's define the front end component that should update itself with help of server-sent events. In case of our app, we choose to add some text below the app title. For example, while the request to the OSM API is running, we want to display the text "Fetching Ice Cafes...":

By displaying this text, the user knows that our app is still waiting for data.

First of all, let's define the DOM element that should receive dynamic updates.

(<h6> '((id . "icecream") "red")) )

With this we define a h6-element with the id "icecream" and the CSS class "red" (which simply prints red text as defined in the pre-installed lib.css) using in-line CSS styling. (Read here for more on in-style CSS).

Now we connect this element to receive server-sent event updates, by using the serverSentEvent function:

(serverSentEvent "id" 'var . prg)

The first argument should be the ID of a DOM component in the page. (...) The ID should be unique within this program session for all serverSentEvent use cases, as it is also used as a key to an internal association table.

The second argument var is a variable which is automatically bound to a socket as soon as the client establishes the event stream connection, i.e. shortly after the page with the embedded call to serverSentEvent is displayed. In essence, this socket is then connected to the DOM component "id".

When that happened, the prg body is executed to set up everything needed for further event sending via the socket in var. var is automatically set to NIL, and the socket closed, when the client closes the connection. Then the server should clean up whatever is necessary.

Basically this means that we need to define a global variable for the var parameter which is used to establish the connection. Let's create a new global called *Sse and add it to our globals list:

(local) (*Sse *IceCreamOsm *Latitude *Longitude *IceList *SupermarketList)

Now we can call serverSentEvents somewhere on our page. The position is not really relevant, but it might sense to keep it close to the connected DOM element to keep the code readable. Since we just want to pass-through some simple status text, we do not need to define any prg. So all we need is to add this line:

(serverSentEvent "icecream" '*Sse)

Defining the server side

The counter part to serverSentEvent is serverSend. As the name already says, serverSend defines when and which data is sent to the client.

serverSend may be called anytime while the connection is open:

(serverSend 'sock . prg)

It sends all output (HTML text) generated by prg to the inner HTML of the component connected to the socket sock.

Let's use it to print out error messages if GPS is not enabled or available, and otherwise the GPS data of the current location.

The task function is checking the position every 6 seconds. Within this check, let's generate a string that is sent to the *Sse socket linked to the icecream component. This string is then pushed to the client and displayed without updating anything else.

(task -6000 1000
   (nond
      ((location?)
         (serverSend *Sse ,"Please enable \"Location\"") )
      ((gps)
         (serverSend *Sse ,"Waiting for location") )
      (NIL
         (setq *Latitude (car @)  *Longitude (cdr @))
         (serverSend *Sse (prin (lat *Latitude) ", " (lon *Longitude))) ) ) ) )

Note: The comma , is a read-macro in case you want to add multi-language support. If you only use one language, it is not needed.

Likewise, we can also add a server-sent update each time any of the buttons is pressed. Since the button press calls a function icecream (defined in lib.l), we can call serverSend there:

(de iceCream (Msg Osm Category Type)
   (serverSend *Sse Msg)
   (ssl "overpass-api.de"
      ...

where Msg is specified in the button press:

(gui '(+Style +Button) "button-icon bg-white mb-5 mx-3"
   T "icecream/img/supermarket.png"
   '(setq *SupermarketList (iceCream ,"Fetching Supermarkets ..." *IceCreamOsm "shop" "supermarket")) ) )

With these small modifications, we receive a much more user-friendly app that gives a minimum of feedback of what is internally going on.

Wrap-up

With server-sent events, we have an elegant and light-weigth way to update some DOM components dynamically without any JavaScript.

Note that unfortunately the map itself cannot be updated this way, because it calls some more complex JavaScript functions that are not updated like this. So if you want to have a dynamic change on the OpenStreetMap (like required for navigation etc.), you have to reload the whole page, for example with the auto function.

The source code for this example can be found here and this is the zip file for PilBox.

Sources

• https://picolisp.com/wiki/?ServerSentEvents
• https://en.wikipedia.org/wiki/Server-sent_events

• https://picolisp-explored.com/an-openstreetmap-app-written-in-picolisp-pt-1
• https://picolisp-explored.com/an-openstreetmap-app-written-in-picolisp-pt-2

At the end of the last post, we had a little app that displays our current location on Open Street Maps. Now we want to find all icecream shops nearby with help of the Open Street Map API.

Working with the Open Street Map API

OpenStreetMap has a convenient API that lets you search for specific facilities. There is also a frontend on https://overpass-turbo.eu/ to play around and understand how it works. For example, this query returns all drinking water positions:

If you navigate to "Export", you can download the raw query data, which looks like this:

node
  [amenity=drinking_water]
  (52.50882953708777,13.415980339050291,52.53089536100559,13.46125602722168);
out

The actual query is surrounded by node and out. The first parameter is [amenity=drinking_water], where amenity is the "key" and drinking_water is the "value". Besides "drinking_water", there are of course also other amenities we can look for which are listed here. There is also an amenity called "Ice Cream" - exactly what we are looking for! The second parameter in the example is the area we are searching, defined by the North, East, South and West borders.

With this information, we can already form a little CURL request to play around with the queries:

 curl 'https://overpass-api.de/api/interpreter?data=node%5Bamenity=ice_cream%5D(52.559290421668074,13.387999534606934,52.580522509085554,13.416066169738768);out;'

And we get back an xml-file with one ice cream shop per node:

<?xml version="1.0" encoding="UTF-8"?>                                                              
<osm version="0.6" generator="Overpass API 0.7.57 93a4d346">                                        
<note>The data included in this document is from www.openstreetmap.org. The data is made available u
nder ODbL.</note>                                                                                   
<meta osm_base="2022-05-18T09:40:07Z"/>                                                             

  <node id="753361477" lat="52.5699404" lon="13.4106207">                                           
    <tag k="addr:city" v="Berlin"/>                                                                 
    <tag k="addr:country" v="DE"/>                                                                  
    <tag k="addr:housenumber" v="4"/>                                                               
    <tag k="addr:postcode" v="13187"/>
    <tag k="addr:street" v="Berliner Straße"/>
    <tag k="addr:suburb" v="Pankow"/>
    <tag k="amenity" v="ice_cream"/>
    <tag k="contact:facebook" v="https://www.facebook.com/Eisspatz/"/>
    <tag k="cuisine" v="ice_cream"/>
    <tag k="name" v="Eisspatz"/>
    <tag k="opening_hours" v="Sa-Mo 12:00-20:00; Tu-Fr 12:00-19:00"/>
  </node>
  <node id="2936989595" lat="52.5682129" lon="13.3993821">
    ...
  </node>
   ...
 </osm>

Now we don't only want to find ice cream shops, but also supermarkets (because these might sell icecream, too). The key-value pair for supermarkets in the OpenStreetMap API is [shop=supermarket]. With this we can get the data we need, but there is still some work to do:

1. Call the API from within PicoLisp,
2. Convert the XML data into a nicer data format.

Calling the API from PicoLisp

This task could actually be easy - simply use the call function to invoke a system command and use curl: (call "curl" "https://<URL>"). Unfortunately, it doesn't work, because the Android system does not give our PilBox app permissions to execute curl.

Luckily we can use the built-in ``ssl``` function for this as well. The syntax is easy:

(ssl 'host 'path . prg) -> any
Executes prg in an input stream (using in) from "@bin/ssl" requesting path from host.

The host is "overpass-api.de" and the path is /api/.... So our request looks like this:

(ssl "overpass-api.de"
   (pack 
      "/api/interpreter?data=node++[amenity=ice_cream]++("
      <Location-Borders>
      ");out;" )
   ( <what to do with the data> )

How do we get the four borders of our map? This is actually easy because it is already tracked within the app. In order to use this data, let's define another global variable *IceCreamOsm:

(local) (*IceCreamOsm *Latitude *Longitude)

and set the OpenStreetMap *Osm variable to it (for example within start):

(start
   (scl 6)
   (setq *Osm '*IceCreamOsm)
   (task -6000 1000
      ...

If you now type *IceCreamOsm in the pty REPL, you will see a list with four values (representing North, South, West, East) and the map's zoom factor.

icecream: *IceCreamOsm
-> ((142536613 . 142609643) (193327006 . 193468454) . 12)

Now all we need to do is to convert these data back to the format that the overpass API is expecting. This is simple math as explained in this post: subtracting 90° from the latitude and 180° from the longitude, and format the output according to the scale. For example, we can get the formatted North coordinate with this:

(format  (- (caar *IceCreamOsm) 90.0) *Scl )

However we should also catch the case that *IceCreamOsm is empty because the map is not drawn yet. Let's add a fallback solution so that we canstill fetch data in the background while the map is loading.

(format
   (- (or (caar Osm) (- *Latitude 0.1)) 90.0)
   *Scl )

This sets the North boundary to 0.1° distance from the current location, which corresponds roughly to a 11 km radius around the current location.

Converting from XML to list format

Now we need to format the XML output on based on our needs. As we could see above, the output looks somewhat like this:

  <node id="753361477" lat="52.5699404" lon="13.4106207">                                           
    <tag k="addr:city" v="Berlin"/>                                                                 
    <tag k="addr:country" v="DE"/>                                                                  
    <tag k="addr:housenumber" v="4"/>                                                               
    <tag k="addr:postcode" v="13187"/>
    <tag k="addr:street" v="Berliner Straße"/>
    <tag k="addr:suburb" v="Pankow"/>
    <tag k="amenity" v="ice_cream"/>
    <tag k="contact:facebook" v="https://www.facebook.com/Eisspatz/"/>
    <tag k="cuisine" v="ice_cream"/>
    <tag k="name" v="Eisspatz"/>
    <tag k="opening_hours" v="Sa-Mo 12:00-20:00; Tu-Fr 12:00-19:00"/>
  </node>

This is a lot of information we don't actually need. Let's keep it simple and only extract the shop name and the GPS position in the following format: (<name> <latitude> . <longitude>). This will form a list similar to this one:

(("Delabuu" 142545086 . 193362049) ("Eiscafé Gelato degli Angeli" 142587148 . 193401989) ("Eisspatz" 142569940 . 193410621) ("Tribeca" 142537430 . 193421053) ("Eis Henri" 142545990 . 193390389) ("Hijís Gelateria" 142551831 . 193413717) ("Eis & Back" 142551954 . 193366761) ("Eislabor" 142541762 . 193422135) ...)

In order to work conveniently with XML data, we can import the xm.l library in the `once`` function.

(once
   (load "@lib/xm.l") )

Similarly to the json-library, the xml function takes XML data and provides some functions to transform it to a list. First we check if the xml output is valid (xml?), and if yes, we travel through the whole XML-tree and visit each valid node.

(when (xml?)
   (for L (xml)
      (when (== 'node (car L))

For each valid node, we want to take the lat and lon attributes from the root as well as the name attribute, and cons these three items to a list. Then we form a final output list with make.

(make 
   (cons
      (attr
         (find '((X) (= '(k . "name") (caadr X))) (cddr L))
         'v )
      (+ 90.0 (format (attr L 'lat) *Scl))
      (+ 180.0 (format (attr L 'lon) *Scl)) )

Fetching the data on button press

We still need to define when the data should actually be fetched. Let's put the ssh function from above into a function iceCream and define it in a separate file lib.l to keep our code more readable. Also we can replace the API variables by local variables:

(de iceCream (Osm Category Type)
   (ssl "overpass-api.de"

where Osm is the current map data and Category and Type are the key/value pairs from the Overpass API. Then we load the lib.l file in the once function.

(once
   (scl 6)
   (load "@lib/xm.l" "icecream/lib.l") )

Next, we define the output lists as *IceList (for ice cream shops) and *SupermarketList (for the supermarkets):

(local) (*IceCreamOsm *Latitude *Longitude *IceList *SupermarketList)

Now we can add the iceCream function as action to our button and store the output in the global variables.

(gui '(+Style +Button) "button-icon bg-white mb-5 mx-3"  T "icecream/img/ice.png"
   '(setq *IceList (iceCream *IceCreamOsm "amenity" "ice_cream")) )

(gui '(+Style +Button) "button-icon bg-white mb-5 mx-3" T "icecream/img/supermarket.png"
   '(setq *SupermarketList (iceCream IceCreamOsm "shop" "supermarket")) ) )

Displaying the data on the map

Finally we can display the data in our map by looping over the *IceList and *SupermarketList respectively and add a little icon and the shop name with the <poi> function.

(for Ice *IceList
   (<poi> (cadr Ice) (cddr Ice)
      "icecream/img/ice-mini.png" "0.1" "1.0" (car Ice) 0 "red") )

Unfortunately the Overpass API sometimes seem a bit unreliable. It can take quite a long time to receive back the data, and sometimes there is no valid data at all. In the last (and final) post of this series, we will add Server-Side Events to let the user know what is going on.

You can find the final code for this app (including server-sent events) under this link and the zip file here.

Sources

• https://software-lab.de/doc/index.html

The app's purpose is simple: It should pick up the current location via GPS and display all ice cream shops and supermarkets nearby, based on the results from the OpenStreetMap API.

Setting up the Working Environment

First of all, let's set up a working environment that allows us to test and modify our code in a fast and efficient way. In order to do this, we will use the PilBox Pseudo-Terminal.

First of all, let's create a simple template. It contains a file App.l which defines the app title as well as the app logo and some basic CSS files. You can find the template here as a reference. Compress the template folder to a .zip-File and send it to the PilBox app on your phone, for example via termux share (or email).

After opening the zip-file in the app it should look like this:

Next we expand the .zip-File in the working directory on our PC and start the pty from the same directory. This way we have the same relative paths and are able to use pbPut and pbGet to copy files.

$ cd <your favorite picolisp directory>
$ unzip iceCreamApp-Template.zip
$ pil ~/pil21/bin/pty <your phone's IP>
android:

If you have trouble with the tty, read here.

Note that PilBox should be open on the phone for the tty to work. After we got the connection, we can check the app folder on the phone:

android: (call "ls")
App.l    JAVA    PID      RPLY  Version  favicon.ico  lib      loc   src
BOSS     LISP    PIL-369  RQST  bin      icecreamPt1  lib.css  log   steps.rc
COPYING  Locale  Port     UUID  db       img          lib.l    log-  tls

This shows the folder icecreamPt1 which includes the template files of the app, as well as PIL-369 which is the internal app's name on my phone (unfortunately my Samsung smartphone seems to rename .zip-files before opening them in external apps).

Now for testing purposes, let's rename the app in App.l from "PicoLisp App Stub" to "Mia' Ice Cream Finder" on the PC and transfer it to the phone like this:

android: (pbPut "<appname>/App.l")

In this example, the appname is "icecreamPt1" because this is how I called it in the repository (link in the end).

Now the app title should be changed on your smartphone as well.

Defining the general app layout

I like starting with the easy things, like setting the header and main layout features. Let's define a function work that gets called. Inside work, we call pilbox, which allows us to define the App header by ourselves instead of the pre-defined "PilBox" text.

"Mias Ice Cream Finder (Pt. 1)" 

(de work ()
   (pilbox
      (<h5> "fh" "Mia's Ice Cream Finder") )
   (menu "Mia's Ice Cream Finder") )

(work)

Note: the CSS-class fh is defined in the pre-installed @lib/phone.css.

Next, we can define the background and styles with bootstrap (here for more info). We display a title called "Want ice cream?" and a search box with two buttons. The buttons can be pressed, but there is no function tied to it yet.

This is the PicoLisp code for that:

(de work ()
   (pilbox
      (<h5> "fh" "Mia's Ice Cream Finder") )
   (menu "Mia's Ice Cream Finder"
      (form NIL
         (<div> "bg-light container-sm border"
            (<div> "d-flex flex-column "
               (<h3> "mt-5 mb-4 d-flex justify-content-center" "Want ice cream?" )
               (<div> "mb-3 d-flex justify-content-center"
                  (<p> NIL "Find ice cream near your current location!") )
               (<div> "d-flex justify-content-center"
                  # Search button for ice cafe
                  (gui '(+Style +Button) "button-icon bg-white mb-5 mx-3"
                     T "icecreamPt1/img/ice.png" NIL)
                  # Search button for supermarket
                  (gui '(+Style +Button) "button-icon bg-white mb-5 mx-3"
                     T "icecreamPt1/img/supermarket.png" NIL) ) ) ) ) ) )

and this is how it looks like:

Getting the current location

Now comes the interesting point: Let's take our current position and display it on OpenStreetMap. First of all, we import the relevant name spaces, some global variables and the gis.l library:

"Mias Ice Cream Finder (Pt. 1)" "@lib/gis.l"

(symbols 'icecream 'gis 'android 'pico)

(local) (*Latitude *Longitude)

Now we can fetch the current position with the gps function, just like in the previous post on GPS. But this time we want to get the data in the background: let's refresh the location every 6 seconds. Periodic tasks can be realized with the task function (here you can more information on task).

(start
   (scl 6)
   (task -6000 1000
      (nond
         ((location?) NIL )
         ((gps) NIL )
         (NIL
            (setq *Latitude (car @)  *Longitude (cdr @)) ) ) ) )

What does this do?

1. First, it sets the scale for integer calculations to 6 (if you don't know the scl function, check out this post).
2. Then we define a periodic task, executed every 6 seconds. The first time it is executed one second after starting the app.
3. nond is a multi-way conditional which executes if the argument evaluates to NIL. We distinguish three cases:
   • (location?) is false, i. e. GPS permissions are not given,
   • (gps) is NIL, i. e. GPS data is not (yet) available
   • NIL: the exit condition - GPS data is available and stored in the global variables *Latitude and *Longitude.

Let's test it: Now the app asks for GPS permissions at start up. If we confirm with "yes", we can execute (gps) in the tty and see the current location:

icecream: (gps)
-> (142520664 . 193401665)

side note: NOT my exact location 😄

Nice! So let's use this data and display it on the map.

(<div> "map"
   (<osm> *Latitude *Longitude 12)
   (<poi> *Latitude *Longitude
      "icecreamPt1/img/here.png" "0.1" "1.0"
      "" 0 "black" ) ) ) ) )

In the next post, we will see how to use the OpenStreetMap API to fetch the supermarket and ice cream shop position and display them in the map. Also, we will use ServerSideEvents to display some information.

You can find the source code up to this point here and the zip-File here.

Sources

• Ice Cream Icon by https://www.irasutoya.com/2015/07/3.html

(This example was also included in the pil64-version of PicoLisp - if you have it installed, you can find the source code in the misc/ folder of your installation.)

How it works

The requirements to our app are very simple: We want to open a TCP port which is dedicated to communication. Any client can connect to this port, for example with telnet, and send and receive messages.

The little GIF below illustrates how the final app should work:

• All "chat users" are connecting to port 4004 in localhost.
• If a new user joins the "room", the other users receive a notification.
• If a user sends a message, the others receive it together with the name prompt.
• If a user leaves the room, the other users are notified too.

File Descriptors and Forks in PicoLisp

All users are connecting to the same port (localhost:4004), where each one is creating an individual socket to communicate with each other. So let's take a look at socket handling in PicoLisp now.

We want the port 4004 to stay open constantly for any client to connect. At the same time, everyone connecting to the port receives their own socket for communication. First, we create a new TCP port with the port function, which returns a file descriptor (read here for more on Unix file descriptors: Everything is a file).

(setq *Port (port 4004))

Now that the port is open, the client can connect. However, we want to be able to handle multiple clients. How can we do this? We use another important Unix feature, which is forking).

A fork creates a copy of a process - a so-called child-process. Now we establish the following division of tasks: The parent process takes care of keeping the port open for clients to connect, while each child process inherits a socket for the client.

We realize this with a loop function. The loop from the parent's process view looks like this:

1. Waits until a client connects with (listen *Port). If a client connects, a new file descriptor is created which is stored in the variable *Sock.
2. Then the process is forked and thus a child process is created.
3. The parent process closes the socked ((close *Sock)) as it is now handled by the child process.

On the other hand, the child process as a copy of the parent is also stepping inside the loop, but instantly leaves the loop to take care of the socket handling. In PicoLisp, we can create a child process with (fork). fork returns NIL in the child and the child's process ID in the parent.

These four lines take care of the parents/child process handling:

(loop
   (setq *Sock (listen *Port))
   (NIL (fork) (close *Port))
   (close *Sock) )

The line (NIL (fork) (close *Port)) corresponds to the exit condition of the for-loop: If for is NIL (i. e., is a child), then the loop is left. If it is non-NIL, it means that we are in the parent process, so we will simply close the *Sock socket and start listening on the port again.

Registering the user name

Now our child process has left the loop and is ready to interact with the user. First of all, we want to register the name. In order to print to the socket's output stream (like telnet), we use the out <FD> function, where is the file descriptor of the socket.

(out *Sock
   (prin "Please enter your name: ")
   (flush) )

(flush) empties the buffered data by sending all to the output stream (= flushing). Alternatively, this can be triggered by a new line, i. e. by using prinl instead of prin.

Next, we read the username from the terminal with (in <FD>). It takes the whole line as input.

(in *Sock (setq *Name (line T)))

Note: In a "real" app, you might want to add some input validation for the name variable.

Spreading the word with tell

Now let's announce to all connected clients that the new user has arrived. This kind of inter-process communication can be handled by tell.

tell is a function that "knows" all process family members and can send them messages (family members are all children of the current process, as well as all other children of the parent process). It takes a symbol as argument specifying the exact message to be sent.

In our case, we want the message to print a text to the current output of the socket. In order to keep it flexible, let's maintain the text itself as a list.

(de chat Lst
   (out *Sock
      (mapc prin Lst)
      (prinl) ) )

chat takes a list as argument, prints each list item and then a line feed. This can now be used within tell:

(tell 'chat "+++ " *Name " arrived +++")

Everytime a new client connects to a port, all other connected clients will be informed with this text.

Handling the actual chat messages

Now finally, we come to use the task function that we've been talking about in the beginning. We create a task, i. e. waiting at the socket for user input. If there is input to this socket, we read it and directly hand it over to tell. Then we wait again for user input.

(task *Sock
   (in @
      (tell 'chat *Name "> " (line T))

However, there is one exception: If the client ends the connection (for example with Ctrl-] in telnet), we spread this information with tell and then close the task. So, the complete task looks like this:

(task *Sock
   (in @
      (ifn (eof)
         (tell 'chat *Name "> " (line T))
         (tell 'chat "--- " *Name " left ---")
         (bye) ) ) )

With this, we have already everything we need for our little chat app. As a last (rather cosmetic) action, we call (wait) in order to surpress the PicoLisp prompt when the chat function gets called.

You can download the full source code of this example here.

Sources

• https://en.wikipedia.org/wiki/Fork_(system_call)
• https://en.wikipedia.org/wiki/Everything_is_a_file
• https://software-lab.de/doc/index.html

Realizing repetitive tasks with task

With help of task, we can schedule tasks to be executed. This can be useful in many situations, like when we want to fetch data periodically from the server, or if we are waiting for an input stream via a socket.

We check the docs:

(task 'num ['num] [sym 'any ..] [. prg]) -> lst

A front-end to the *Run global. If called with only a single num argument, the corresponding entry is removed from the value of *Run. Otherwise, a new entry is created. If an entry with that key already exists, an error is issued. For negative numbers, a second number must be supplied. If sym/any arguments are given, a job environment is built for the *Run entry.

A simple counter

This is a lot of information, so let's look at the most straightforward example first: Printing a counter 1, 2, 3, 4, 5.... We want to print one number per second (i. e. 1000 ms), and we want to start immediately (i. e. after 0 ms).

The syntax for this is as follows:

: (task -1000 0 (tty (println (inc (0)))))

The first number defines the frequency, the second number defines the delay until the command is first executed. The first number needs to be negative.

• Note 1: we use (tty (println ...)) instead of only println, because this will keep the command prompt in the REPL stable, as you can see in the following gif:

Without tty: The command prompt gets overwritten by the println-output, the result is very messy.

• Note 2, for PicoLisp enthusiasts: (inc (0)) destructively increments the cell which initially contains 0. For this reason, the return value will constantly increase. Note the difference to (inc 0), which will always return 1.

Now how can we stop the counter? As the documentation tells us, the first number servers as identifier. This means in order to stop this task, we can simply call:

(task -1000)

The *Run global variable

As the docs tell us, task is only a front-end to the global variable *Run. So let's execute task and check the content of *Run to understand what that means.

If no task is running, *Run is NIL:

: *Run
-> NIL

Now let's set a task and try again:

: (task -10000 -10000 (tty (println (inc (0)))))
-> (-10000 -10000 (tty (println (inc (0)))))
: *Run
-> ((-10000 -8102 (tty (println (inc (0))))))

We have set a task that fires after 10 seconds and then every ten seconds. If we check *Run, we see that the second item of the list seem to contain the time difference up to the next execution, as well as the current values:

1
2
3
4
5
: *Run
-> ((-10000 -1251 (tty (println (inc (5))))))
6
:

Conditional Execution of task

Now let's extend our little example: We want to count down from 10 to 0, after that the task should be stopped.

task implicitly defines a closure (read here for more on PicoLisp closures). This means we can also execute more complex functions:

• first we set N to 10,
• then we print it,
• then we decrement it with (dec 'N)
• then we check if the result is equal to 0.
• If it is equal to zero, the task is stopped.

In other words:

: (task -1000 0 N 10 (tty (println N)) (and (=0 (dec 'N)) (task -1000)))
-> (-1000 0 (job '((N . 10)) (tty (println N)) (and (=0 (dec 'N)) (task -1000))))
10
9
8
7
...

As you can see, task wraps a job function around the prg part of the task. job first takes a list with the current values of the symbols (in this case (N . 10)), and then the program body which defines the new values of the symbols.

Working with file descriptors

There is a second way to work with task: Instead of a periodic time in milliseconds, we can define a file descriptor. File descriptors are unique identifiers for files or other input/output resources such as pipes or networks in Unix systems ("File descriptors" on Wikipedia).

The file descriptors 0, 1 and 2 are defined as Standard Input, Standard Output and Standard Error. After that, each new file or socket we open will define a new file descriptor, which is also the return value for the picolisp functions like open and port.

: (open "myfile.txt")
-> 3
: (port 4444)
-> 4

Note: port 4444 opens a TCP port on 4444. For an UDP port, the second arguments should be T: port T 4444.

Now what can we do with this? We can define a task which "listens" on a port and executes all commands that it receives:

: (task (port T 4444) (eval (udp @)))
-> (3 (eval (udp @)))

(udp <FD>) receives one UDP package at the corresponding file descriptor, which is set to the socket at port 4444 in the example above.

Next, we can send data to this socket via

: (udp "localhost" 4444 '(println *Pid))  # Send RPC message
-> (println *Pid)

and it will be evaluated because of the eval function defined in the task.

In the next post, we will go a little bit deeper into the socket handling with task by exploring a very simple messenger app.

Sources

• https://en.wikipedia.org/wiki/File_descriptor
• https://software-lab.de/doc/index.html

But before we dive into Open Street Map, we need to understand how to interact with the GPS module of our smartphone.

Importing the gis.l library

In order to use the OpenStreetMap and Google Map functions, we need to import the @lib/gis.l library. gis.l contains some GPS-specific functions such as defining latitude and longitude, calculating the distance between two coordinates and the interaction with the OpenStreetMap and Google Maps API.

We can import additional libraries by adding them to our App.l file directly after the app name declaration. Also we need to add gis to the App's name space in order to make the functions from gis.l available.

"Mia's Ice Cream Finder" "@lib/gis.l"

(symbols 'icecream 'gis 'android 'pico)

Checking for permissions

Just like in the camera example, we have to verify that we actually have permission to access the phone's GPS. Just like there is a camera? function to check the camera permissions, there is a location? function to check the GPS permissions.

If GPS is not permitted, we could display a little warning "Please enable 'Location'":

(unless (location?)
   (<p> "tiny red"
      ,"Please enable \"Location\"" ) ) )

location? is defined in assets/run/lib/android.l.

Latitude and Longitude - some basics

GPS coordinates are represented in degrees, relative to the equator for the North/South coordinates (latitude), and relative to the prime meridian for West/East (longitude).

The latitude domain ranges from -90° (South Pole) to +90° (North Pole), and the longitude domain from -180° to 180°. Both -180° and +180° are indicating the same location, which is the "anti-meridian".

For example, Berlin is located at (52.52°, 13.404°). Since it is on the North-Eastern quarter of the world, both values are >0.

Getting the current location

We can get the current location with the (gps) function. You can easily test it if you get a pseudoterminal from your PC to the PilBox and type (gps) (of course, GPS permission must be available):

icecream: (gps)
-> (143571890 . 192396667)

The function returns two integer values which stand for the Latitude and Longitude. However, the values are not represented in the same way as we normally expect. First of all, we need to fix the scale, as PicoLisp only uses Fixed-Point Calculation.

The scale is 6, which means that the formatted GPS output looks like this:

icecream: (format (car (gps)) *Scl ))
-> "142.521890"
icecream: (format (cdr (gps)) *Scl )
-> "193.404667"

Secondly, the latitude domain range at PicoLisp is mapped from [-90, 90] to [0, 180]. Likewise, the longitude domain range is mapped from [-180, 180] to [0, 360]. This simplifies the internal representation. So in order to re-transform the values, we need to subtract 90 degrees from the latitude and 180 degrees from the longitude, and adjust the scale.

icecream: (format (- (car (gps)) 90.0) *Scl )
-> "52.521890"
icecream: (format (- (cdr (gps)) 180.0) *Scl )
-> "13.404667"

Let's get the current location with a button which adds both values to the global variables *Latitude and *Longitude.

(gui '(+Able +Button) '(location?) "Get my location"
   '(let G (gps)
      (setq *Latitude (car G)  *Longitude (cdr G)) ) )

Displaying the current location on a map

There are two main choices if you want to use a map in your app:

• Google Maps, which is arguably the bigger one, but it requires an API-Key and is not free (depending on the number of requests). You can read more about it on the Google Developer docs. Of course you can embed an iframe of a map showing the current location, but the user will not be able to interact with it.

• The alternative choice is Open Street Map, which is entirely free, but the content might be less up-to-date and the data quality depends a lot on the number of volunteers who provided the data.

Variant 1: Google Maps

Google maps can be simply integrated with the <google> function which takes four arguments: Latitude, Longitude, Width and Height. For example:

(<google> *Latitude *Longitude "100%" 400)

produces a map which fills the current container's width and 400 px height.

If you check the source code of <google> in @lib/gis.l, you will see that this simply maps all values to an iframe:

(de <google> (Lat Lon DX DY)
   (prinl
      "<iframe width=\"" DX "\" height=\"" DY "\" frameborder=\"3\" \
      src=\"https://www.google.com/maps?source=s_q&amp;q="
      (fmt Lat "," Lon)
      "&amp;output=embed\"></iframe>" ) )

Variant 2: Open Street Map

Open Street Map can be integrated with the <osm> function, which takes three arguments: Latitude, Longitude and the zoom level, which is defined in the Open Street Map documentation. A zoom level of 12 corresponds to a "town, or city district".

(<osm> *Latitude *Longitude 12)

Additionally, you can also pass more parameters defining the click and update behaviour, but it is not needed for our minimal example.

The nice thing about Open Street Map is that the developer is free to add additional icons or text to the map. For example, for our Ice Cream app, we can mark all ice cream locations with an ice cream icon, which is not possible in Google Maps.

This can be done via the <poi> (point of interest) function, which takes as a minimum latitude, longitutde, image, an x and y offset, and a text with a text offset.

For example, these lines add a little pen icon and the word "Here" to the specified position on the map:

(<poi> *Latitude *Longitude
   "@img/go.png" "0.1" "1.0"
   "Here" -20 "black" )

With this, we can already write a little app that displays the current position on a map.

Next, let's try to retrieve the position of all listed ice cafes from the Open Street Map API and try to display it in our app.

Sources

• https://commons.wikimedia.org/wiki/File:Latitude_and_Longitude_of_the_Earth.svg
• https://software-lab.de/demo.zip
• https://wiki.openstreetmap.org/
• https://developers.google.com/maps?hl=de

In this post, we will finalize the app by adding cache deletion, CSS and name spaces.

Deleting the cache

If you tested the previous version of the app, you might have noticed that the picture does not refresh. This is because the picture is loaded from the cache of the app and is not refreshed.

Luckily, it is very easy to solve it: we only need to add the (clearCache) function somewhere in our code, for example before reloading the page.

'(takePicture (tmp "img.jpg")
   '((Intent)
      (clearCache)
         ...

Adding CSS

Next, let's add our own CSS files to our app. The PilBox searches for a file called lib.css in our app's home folder. So we have two possibilities: We could either write all our CSS specs into lib.css, or simply import other CSS files into lib.css.

I think the second approach is better, because it allows to use multiple CSS-files - for example, a bootstrap layout and an app-specific one:

@import "css/bootstrap.css";
@import "css/camera.css";

I like using the Bootstrap pre-defined containers because it saves a lot of time in the development process. Here you can read how to add CSS classes to PicoLisp GUI components.

Adding icons

We can also add a dedicated icon to our app. It has to be saved to a file called logo.png. Additionally, we can also replace the "take Picture" button by an icon and define it's size and behaviour in our dedicated CSS file:

(gui '(+Style +Able +Button) "button-icon bg-white" '(camera?) T "miasCamera/img/camera.png"

where the class button-icon is defined as:

.button-icon {
   width: 60px;
   height: 60px;
   padding: 5px;
   border: 1px solid #ddd;
   ...
   border-radius: 6px;
   ...
   box-shadow: 0 3px 5px #ccc;
}

.button-icon:hover {
   margin: 5px auto 3px;
   ...
}

which gives it a more button-like look & feel when it is pressed.

Setting the picture size

We also want a responsive size of our picture. It should be centered in the middle and its width should not be larger than the smartphone width.

For this, we can simply add a width: 80%; height auto attribute to the image class, which will take 80% of the parent's container size.

(<div> "d-flex mb-5 justify-content-center" (gui '(+Style +Var +Img) "my-img" '*Picture ))

where CSS is

.my-img {
   width: 80%;
   height: auto;
}

After these modifications, the app should look like this:

Local variables and name spaces

For the sake of readability, we could now introduce a local variable *Picture instead of the full path when we want to address the file.

However, there might be several apps running in parallel in our PilBox app, and maybe other apps also have a local variable called *Picture. So let's make sure that these don't get mixed up by adding our own local name space to the app:

(symbols 'miasCamera 'android 'pico)
(local) *Picture

This means that symbols are first searched in the miasCamera name space, then in android, and finally in pico which is the "standard" PicoLisp namespace.

Now we can set our global variable without worrying about name conflicts.

(setq *Picture "miasCamera/img/img.jpg")
...   
   (gui '(+Style +Var +Img) "my-img" '*Picture )
...

Adding the work and start functions

Our app works now without problems. However, it is cleaner to wrap the code to be executed each time the app is opened into a work function, and the code that should only be executed once at start of the app in a start function.

However, in this specific case, the start function is empty because there are no specific things to be loaded.

(start)

(setq *Picture "miasCamera/img/img.jpg")

(de work ()
   (menu "Mia's Camera App"
      ... ) )

(work)

With this, our very first (and not too useful) PicoLisp app is finished!

You can download the source code of the finished app here and the zip-file here.

Sources

• http://software-lab.de

Now I would like to start a little series of small independent "Apps". We will start with a little "Camera" app. The functionality is extremely simple: We can take a picture and save it to the app's local storage. If we take another picture, the old one is overwritten.

Check Camera Availability

First of all, we should investigate how we can address the smartphone camera from within our app. For a little inspiration, let's check the camera.l file in the demo app.

The permissions flow as defined in the android developer tools is as follows:

Step 1 (declaring the permission in the app's manifest file) is already done by the PilBox APK. Let's focus on the flow from step 3 to step 6: Checking for available permissions and ask for setting them.

We can test if the camera is enabled with the (camera?) function. It checks if the camera is available for the smartphone and if the camera permission is set. If permission is not available, the permit function asks for permission by launching the typical Android pop-up:

Let's create an App.l in our new app and put the following code inside:

(menu "Camera"
   (form NIL
      (gui '(+Able +Button) '(camera?) "Take Picture")

This will simply render a button that is enabled if the camera permission is already set. Otherwise, permission will be requested for the user.

Optional: Check the sources

Theoretically, it is enough for us to know that camera? does what we want: Check permissions and get them if needed.

However, I think it's worth to also study the PilBox source code to see how this exactly maps to the classes and methods on https://developer.android.com.

camera? checks if the camera is available for the smartphone and whether the camera permission is set. Otherwise, the function returns NIL. You can find it in line 286 in assets/run/lib/android.l.

(de camera? ()
   (and
      (java (java CONTEXT 'getPackageManager) 'hasSystemFeature "android.hardware.camera")
      (permit "android.permission.CAMERA") ) )

The and condition checks two things: First, whether the smartphone actually has a camera, and secondly, if the permissions are enabled. If not, the permit function asks for permission. How does it work?

As explained in the post Intents and the Java Interface, the first condition of camera? maps directly to the Java method

getPackageManager().hasSystemFeature("android.hardware.camera")

The second condition calls the permit function which is defined in line 123 of assets/run/android.l. It connects two steps with an or clause:

• Has the permission already been granted?
• If not, ask the user for permission and return the result.

The permission check is realized by the ContextCompat.checkSelfPermission() method, while the actual permission request is done by ActivityCompat.RequestPermission().

Called from PicoLisp:

(de permit (Str)
   (or
      (=0
         (java  "android.support.v4.content.ContextCompat"
            'checkSelfPermission
            (; CONTEXT GUI)
            Str ) )
      (java  "android.support.v4.app.ActivityCompat"
         'requestPermissions
         (; CONTEXT GUI)
         (list Str)
         0 ) ) )

where Str is the relevant permission, in our case "android.permission.CAMERA".

Taking a picture and storing it

Now let's assume the user has given permission to use the camera. Next, we want to take a picture and store it. For this purpose, we can use the pre-defined function takePicture, which takes a destination and a function as arguments.

In our case, let's store the picture in the tmp-folder of our app and reload the page. The tmp folder is automatically deleted at the end of the PilBox process.

'(takePicture (tmp "img.jpg")
   '((Intent)                           
      (loadUrl (baseHRef) *SesId "miasCamera/App.l") ) )

The tmp function creates a folder in .pil/tmp/<*Pid>/, where *Pid is the current process ID. With this, we create a "take picture" intent for the native camera function. The return value of the intent, is the input value for our anonymous function '((Intent). Strictly speaking, it is not needed because we don't use the intent's return value; instead we simply reload the page. But it helps to illustrate that we are working with an intent here.

Of course, we don't only want to have the picture in our tmp folder, but we want to display it in our app and keep it even if the app is restarted. So let's copy the temporary file to our local app folder and display it:

(gui '(+Able +Button) '(camera?) "Take picture"
   '(takePicture (tmp "img.jpg")
      '((Intent)       
         (call "cp" (tmp "img.jpg") "miasCamera/img/img.jpg")             
         (loadUrl (baseHRef) *SesId "miasCamera/App.l") ) ) )
(gui '(+Var +Img) "miasCamera/img/img.jpg")

With this, we have fulfilled the minimum requirements of our basic app: Asking for camera, taking a picture and displaying it. It is not very usuable yet, so this point will be discussed in the next post.

Deploying the app

How can we get our app now onto our smartphone? There are two ways:

1. Create a zip-File, saving it to the phone and opening it with PiLBox.

This approach works very well with finished apps, but is not so helpful for development, because you end up zipping and transferring all files for each minor change.

2. Get a pseudoterminal on your phone via WiFi and edit the files directly.

This approach has been explained in this post: Getting a shell to your PC. After setting it up as described in the post, you connect to your phone with

~/pil21/pil @/bin/pty <smartphone-IP-address>

(assuming that pil21 is installed in your home folder, otherwise replace the path).

Then you can either modify the source code directly with the vi-based PicoLisp editor vip, or you can edit the files locally and push them with pbPut <folder>/<file>. Note that in this case, the folder structure needs to be the same locally and on your phone.

: (pbPut "miasCamera/App.l")

This allows you to modify the app instantly.

In the next post, we will improve the app's look & feel, as it is barely usable at this point.

The source code of the example up to here can be found here, the zip-File here.

Sources

• http://software-lab.de/

The "Generic App" Source Code

First of all, let's return to the original pilBox.apk that you can download here. Let's unzip it to take a look at the sources. We can find the Android Manifest, the Java classes and a few other documents there.

Note: if you are interested in the complete data (like the original Java sources), you can get it here: PilBox-complete; installation details in the ReadMe.

What we are interested in specifically is the folder assets/run, which specifies the PicoLisp environment. For example, we can find the PicoLisp libraries in assets/run/lib.

Now let's take a look at assets/run/App.l. It is titled "Generic PicoLisp App" and defines the environment of the apps, such as loaded libraries, namespaces and so on.

"Generic PicoLisp App"

(load "@lib/net.l" "@lib/misc.l" "@lib/btree.l" "@lib/db.l" "@lib/pilog.l")

(msg *Pid " " (stamp))

# Generic PilBox App
(allowed ()
   "!pb0" "!pbHome" "!pbRepl" "@lib.css" "@lib/phone.css" "!pb.css"
   "!pbSettings" "!pbList" "!pbCat" "!work" )

...

(symbols (setq *PbSymbols '(android pico)))

...

Now with that in mind, we will check out some of the demo applications.

A minimal example

Let's begin with the hello.zip demo app, which is truly minimal. After unzipping it, we get a folder hello with a single file inside, which is App.l. This is it's content:

"Hello World"

(menu "Hello World!"
   (<h1> "center fh" "Hello, World!") )

The first string is the description title of the list which is rendered to the home screen button.

Since we didn't specify a logo, the standard PicoLisp logo is used.

The menu function is rendered to this:

menu is defined in assets/run/App.l:

(de menu (Ttl . Prg)
   (unless *SesId (forbidden))
   ...
  (action
     (html 0 Ttl
        (if (hasFile "lib.css")
           (list "@lib.css" "@lib/phone.css" @)
           '("@lib.css" "@lib/phone.css" "!pb.css") )
           ...
         ((0 0 'main)
            (prog
               (<div> @
                  (run Prg 1) )
               ...

It basically defines the header with the logo and settings wheel and the HTML and CSS environment for the program to be executed. In our case, the title is "Hello World!" and the program is (<h1> "center fh" "Hello, World!") ) which just prints a string.

Error handling: The crash demo app

Now let's take a look at a slightly more complex example: the "Crash" app. Besides the App.l it contains a file called logo.png which is displayed on the Home screen and in the upper left corner.

By clicking on the "Crash me!" button, you can provoke an error.

(menu "Crash me"
   (form NIL
      (<div> "center"
         (<h2> NIL "Crash me")
         (gui '(+Button) "Crash now!"
            '(/ 1 0) ) ) ) )

The code is defined in the PicoLisp error handler *Err in assets/run/App.l of the PilBox source code (line 361):

(de *Err
   (println '==== *Pid (symbols))
   (for ("L" (trail T)  "L")
      <...>
   (println '====)
   (loadTxt
      "<style type=\"text/css\">body{color:white;background-color:red}</style>"
      "<h1>☹</h1><pre>" (sym ^) "</pre>"
      "<pre>" *Msg "</pre>" )
   (wait 420) )

where *Msg is a PicoLisp global variable holding the last recently issued error message.

"Standard" GUI elements

As long as no smartphone-specific functions are required, we have now all the knowledge to create purely PicoLisp-based apps. Basically the structure will be the same like in the Web Apps tutorials, except that the code is not executed on the server but by the PilBox.

Some good examples for this are in the demo app:

• Canvas: the Forest Fire
• Fields: A collection of HTML text input fields,
• Charts: A simple table,
• Life: A simple animated version of the "Game of Life" algorithm.

In the next couple of posts, we will discuss apps which use the java interface to access device functions such as camera and GPS.

Sources

• https://software-lab.de
• https://software-lab.de/doc/index.html

Often, the interesting part of a smartphone app needs to acces smartphone-specific resources (like camera, GPS..). In this case, we need to leave our PicoLisp environment and send an intent to an app which provides the needed resources. This post will cover a general explanation of intents and how we can call them from PicoLisp using the Java Interface.

Android Intents

Intents are messaging objects that can be used to request an action from another app component such as camera, GPS, contacts and so on. There are three fundamental uses cases:

• starting an activity, which is represented by a single screen in an app,
• services, which run in the background,
• or broadcast deliveries: messages that any app can receive.

There are two types of intents:

• Explicit intents specify which application will satisfy the intent, by supplying either the target app's package name or a fully-qualified component class name.
• Implicit intents do not name a specific component, but instead declare a general action to perform, which allows a component from another app to handle it.

Obviously, implicit intents are inherently less secure because there is no control from developer side which app will accept the intent. The Android system will simply find all appropriate components based on the intent filters in the manifest files of other apps.

From the android docs: 1. How an implicit intent is delivered through the system to start another activity: [1] Activity A creates an Intent with an action description and passes it to startActivity(). [2] The Android System searches all apps for an intent filter that matches the intent. When a match is found, [3] the system starts the matching activity (Activity B) by invoking its onCreate() method and passing it the Intent.

Now let's see how we can call intents from our PicoLisp app.

Example: Camera takePictureIntent as per Android docs

A typical intent is a call to take a picture from the smartphone app. Let's look at the example Java code from the Android documentation.

First, we need to check if the camera is available:

getPackageManager().hasSystemFeature(PackageManager.FEATURE_CAMERA_ANY)

Then we can call the intent, which involves three pieces: The Intent itself, a call to start the external Activity, and some code to handle the image data when focus returns to your activity.

static final int REQUEST_IMAGE_CAPTURE = 1;

private void dispatchTakePictureIntent() {
    Intent takePictureIntent = new Intent(MediaStore.ACTION_IMAGE_CAPTURE);
    try {
        startActivityForResult(takePictureIntent, REQUEST_IMAGE_CAPTURE);
    } catch (ActivityNotFoundException e) {
        // display error state to the user
    }
}

Now let's see how we can call this from PicoLisp.

The java function

Let's take a look at lib/android.l in your local pil21-installation (or in the PilBox source code). Here we find the definition how to access the Java environment from PicoLisp in line 37 to 45:

# Java I/O
# (java "cls" 'T ['any ..]) -> obj         New object
# (java 'obj ['n] 'msg ['any ..]) -> any   Send message to object
# (java 'obj "fld" ['any]) -> any          Value of object field
# (java "cls" ['n] 'msg ['any ..]) -> any  Call method in class
# (java "cls" "fld" ['any]) -> any         Value of class field
# (java T "cls" ["cls" ..]) -> obj         Define interface
# (java 'obj) -> [lst ..]                  Reflect object
# (java "cls") -> cls                      Get class

What does this tell us. The types of the arguments to java specify the type of the call. These types may be

• symbols (transient, internal or external),
• short numbers,
• or T.

(read here for data types of PicoLisp).

Then a transient symbol cls denotes a Java class name, and fld a field in a class:

• An external symbol obj denotes a Java object
• An internal symbol msg denotes a message (method call)

The following any arguments may be of any type (numbers, symbols (normally strings) or lists (e. g., arrays in Java).

Now let's "translate" the line of code which checks the camera permissions:

getPackageManager().hasSystemFeature(PackageManager.FEATURE_CAMERA_ANY)

hasSystemFeature is a method of the Android PackageManager abstract class and getPackageManager is a method of the abstract Context class. So this means, the line above is actually a short form of:

context.getPackageManager().hasSystemFeature(PackageManager.FEATURE_CAMERA_ANY)

Now in PicoLisp: we can express context.getPackageManager() by

(let PM (java CONTEXT 'getPackageManager))

where CONTEXT is a pre-defined database object describing the android context (more on this later) and 'getPackageManager is the message. This corresponds to the syntax:

# (java 'obj ['n] 'msg ['any ..]) -> any   Send message to object

Then we call thehasSystemFeature method of the package manager class:

(java PM 'hasSystemFeature "android.hardware.camera")

Now let's look at the takePictureIntent in Java and PicoLisp:

# Java
Intent takePictureIntent = new Intent(MediaStore.ACTION_IMAGE_CAPTURE);

# PicoLisp
(let Intent (java "android.content.Intent" T "android.media.action.IMAGE_CAPTURE"))

Next, the Java code

startActivityForResult(takePictureIntent, REQUEST_IMAGE_CAPTURE);

can be mapped to the PicoLisp function startActivityForResult which is defined in lib/android.l. Note that the intent is already defined within in the PicoLisp function startActivityForResult:

(de startActivityForResult (Fun Action . @)
   (let Intent (java "android.content.Intent" T Action)
      (catch '("ActivityNotFound")
         ...

where Fun is the function that should be handled and Action is the activity, in this case "android.media.action.IMAGE_CAPTURE". It generates a new object by calling the new() function in Java.

The complete example:

# Java (try-catch removed)
Intent takePictureIntent = new Intent(MediaStore.ACTION_IMAGE_CAPTURE);
startActivityForResult(takePictureIntent, 1);

# PicoLisp
(startActivityForResult Fun "android.media.action.IMAGE_CAPTURE")

where Fun defines what to do with the picture. We will see an example of this in the next post, where the picture will be saved in a temporary directory.

The CONTEXT object

In the example above, we used the CONTEXT object in (java CONTEXT 'getPackageManager) without explaining what is actually is. So let's do that now: CONTEXT is defined in lib/android.l as

# Android Context
(local) (CONTEXT GUI Config javaExt)
(de CONTEXT . {H@@@40000000000})  # file 32768, obj (hex "100000000")

where {H@@@40000000000} is the external database object. We can inspect it in the PilBox Repl or from your local pty-connection:

android: (show CONTEXT)
{H@@@40000000000} ({H@@@1404477010} . "de.software_lab.pilbox.PicoLisp")
   GUI {H@@@1725467726}
   Home "/data/user/0/de.software_lab.pilbox/files"
   Pid {H@@@27043500604}
   Port "8081"
    <...>
   Uuid "de1958ea-d909-44a7-83f7-cd4d750a68ca"
   Loaded T
-> {H@@@40000000000}

As you can see, the object's class is "de.software_lab.pilbox.PicoLisp". Let's check that one:

android: (show '{H@@@1404477010})
{H@@@1404477010} ({H@@@1114542263} . "android.app.Service")
   NIL
   mActivityManager
   mApplication
   <...>
-> {H@@@1404477010}

and we find that the superclass is "android.app.Service" which shows that PicoLisp is actually running as a service. Apps have activities and services - the PilBox activity can be inspected in the GUI property of CONTEXT:

android: (show  (get CONTEXT 'GUI))
{H@@@1725467726} ({H@@@1233152210} . "de.software_lab.pilbox.PilBoxActivity")
   PilView {H@@@46020644}
   <...>
   Config {H@@@1565731746}
   <...>
   Home "/data/user/0/de.software_lab.pilbox/files/"
   <...>
-> {H@@@1725467726}

From here, you could check the Config component with (show (get CONTEXT 'GUI 'Config) or PilView with (show (get CONTEXT 'GUI 'PilView)), and so on.

This post showed a quick (and rather superficial) introduction to how to map Java and PicoLisp code in order to access smartphone components. In the next posts, we will study some examples from the demo applications.

Sources

• https://developer.android.com

Today we will see how we can get a shell on our smartphone with help of PilBox.

How it works

We establish a pseudoterminal (PTY) between our smartphone and our desktop computer or any other device you plan to use. The PTY is established between the ports of the two process and enables asynchronous, bidirectional communication.

In order for this to work, both devices don't have to be physically connected, but they need to be in the same network. It will not work from mobile network, so as first step, connect both devices (phone and PC) to the same WiFi.

Identify your phone's IP address

As first step, we need to identify our phone's IP address. If we had full privileges on the PilBox REPL, we could call ipconfig or ip addr or similar commands to get the IP address, but unfortunately these are blocked. One easy way to find the local IP address is by checking the list of connected devices in your router.

Now, as we learned before, the PTY needs two open ports. In other words, our PilBox is opening a port on your smartphone. You can confirm that for example with the command line tool nmap. The output will probably look something like this:

$ sudo nmap <smartphone-local-ip>
Starting Nmap 7.80 ( https://nmap.org ) at 2022-02-22 17:46 CET
Nmap scan report for <your-smartphone>
Host is up (0.017s latency).
Not shown: 997 closed ports
PORT     STATE SERVICE
8081/tcp open  blackice-icecap

Nmap done: 1 IP address (1 host up) scanned in 21.16 seconds

There is one open port, 8081. (There might also be more depending on what other apps you have installed.) I think the "blackice-icecap" service name is just a wild guess from nmap because PilBox is not in their database yet, so we can ignore that.

Change the port (if needed)

If the port 8081 is clashing with some other apps, you also have the possibility to adjust the port. For this, simply access the file Port (located in the app's home directory) in the REPL by typing "Port" and press "Edit".

After that restart the PilBox and scan again. Now nmap should show two open ports on 9999:

PORT      STATE SERVICE
9999/tcp  open  abyss

Creating the "secret key"

Next, you need to create a "secret key" file in the home directory of both app and your PC. The content of both files should be the same. The file name is .pty. For example, I created a file with content "testkey" in my home directory on my laptop:

$ cat ~/.pty
testkey

and on my phone:

I encourage you to choose a stronger pass file than "testkey".

If you now restart the PilBox, it will run in debug mode, as can be seen because the debug button in the REPL will be disabled, and if you run nmap again, you would notice a second open port, one digit higher then the existing one (default: 8081 + 8082). These two ports will be used for the communication.

Starting the session

The script to establish the PTY can be found in the bin/pty file of your pil21-installation.

Note: If you have changed the port number from 8081 to another port, you need to modify the port variable in the starting script, too (line 7).

Then we can establish the connection. For example, if your pil21-installation is in your home folder and the local IP address of your phone is 192.168.178.27, it looks like this:

~/pil21/pil @/bin/pty 192.168.178.27
-> NIL
android:

Just like in the smartphone REPL, we get a PicoLisp REPL in the android namespace. This is nice, but I promised you a shell. So type: (call "sh"). Now we are on the phone!

android: (call "sh")
:/data/data/de.software_lab.pilbox/files $ whoami 
u0_a384
:/data/data/de.software_lab.pilbox/files $ ls 
App.l    PIL-battery  PIL-radio   Version  db           lib.l  steps.rc
BOSS     PIL-browser  PIL-steps   battery  demo         loc    tls
COPYING  PIL-calc     PIL-wecker  bin      favicon.ico  log    wecker
JAVA     PIL-chess    Port        browser  hello        log-
LISP     PIL-crash    RPLY        calc     img          radio
Locale   PIL-demo     RQST        chess    lib          src
PID      PIL-hello    UUID        crash    lib.css      steps

Note: Be aware that the communication between the devices is not encrypted, so don't use this in networks you don't trust.

Editing files

The shell we get is rather limited, you will not be able to use a text editor. It is better to modify source code files from the REPL. There are two ways to do it:

• use Vip, the PicoLisp editor,
• or modify the files on your local machine and use the commands (pbPut <filename> and (pbGet <filename>) to get and change files.

Using VIP

Let's say we want to check the log files. In this case, call this in the REPL:

: (vi "log")

It will open a vi-like editor (vip) that allows you to modify the file. Likewise, you can use vi for source files and to see function definitions:

: (vi "radio/App.l")
: (vi 'car)

More on Vip can be found in this post.

Using pbPut, pbGet

Two other very useful functions are pbPut and pbGet. Assumingafter you want to edit the string in .pty, you can first get the file to your local machine:

: (pbGet "log")

modify it with your preferred editor, and put it back up:

: (pbPut "log")

Note that you cannot specify the destination folder, so you should start the pty from your local equivalent to the app "home" folder. E.g.: If you want to copy the file myApp/App.l, you need to have a folder myApp both on your local machine and on your smartphone. If it is not available yet, you need to create it:

: (call "mkdir" "myApp")
-> T
: (pbPut "myApp/App.l")

Also, be aware that any exiting files with the same name will be overwritten, which is very convenient for updating source code but can also lead to sad accidents.

Now we're almost ready to write our own apps. The next post will explain how the Java interface can be reached from PicoLisp, and then we will take a look at how PicoLisp apps are structured. This should give us all the knowledge we need to write our own apps.

Sources

• http://software-lab.de/pilBox.apk
• http://picolisp.com

About the PilBox REPL

The PilBox REPL provides us with a PicoLisp interpreter and a very simple shell on our phone. However, typing on the phone can be quite tedious, and it is not adequate for complex operations. A little teaser: In the next post, we will see how to get a proper PilBox shell from your PC.

But for now, let's try to understand how the built-in shell works.

Basic functions

When you open the REPL page (by clicking on the PicoLisp icon in the top left side), you can see a big white field and a small input field below.

Before the input field comes the current namespace, which is currently android. This means that the interpreter first searches the android namespace for a given function and after that the standard 'pico' namespace.

We have two ways to interact with it:

• either we type standard "PicoLisp" code,
• or we can use invoke shell commands, if we prefix them with !.

In the example above, you can see that I checked the current system with the shell command ! uname -a, and executed the PicoLisp code (* 3 4).

In other words, with this we have a simple shell on our current phone. whoami returns the PilBox user, which in my case is u0_a384, and ls -la shows us all the files in the current directory. Here you can see what the "sandbox" concept in Android means: all files in the folder belong to user u0_a384 and no other user (process) has access to it.

Of course it also means that we cannot call commands that require higher privileges: for example (! ifconfig) returns No file /proc/net/dev: Permission denied.

Loading files

Next to the "eval" button, you can find an "edit" button. This button takes the current text in the input field and opens a file of the corresponding name. If no such file exists, a new one is created (but it will only be saved after we have typed something).

Let's try: We could open the main file (App.l) from the "hello"-demo and modify the code. It can be found in the hello/ folder:

Let's tap into the field and change "Hello World" to "Hello, Goodbye". When we click on the top left icon and then on our "Hello" app, we will see that it has changed:

Check the logs

The list of files, viewed with ! ls -l, also shows two interesting files: log and log-. Both of them are log files; while log contains the data of the current process, log- contains the data of the previous process. Accordingly, it can contain important information if your app suddenly stops working.

You can check the content of the files by typing log and press the EDIT-button, or even easier, type ! cat log / ! cat log-.

"Clear Cache" and "Debug"

There are two more buttons: "Clear Cache" and "Debug". When you press "clear cache", you usually won't see much, unless you changed for example the CSS-file: In this case clearing the cache will automatically load the new CSS file (otherwise it might take up to 24 hours).

If you tap the debug-button, you will enter the "debug" mode. This loads all necessary files, and after that the button gets disabled and can't be tapped again. Debug mode is enabled automatically if we connect the smartphone to the computer via a pseudo-tty, as we will see in the next post.

Sources

• www.software-lab.de/pilBox.apk

Luckily, there are already some demo apps available: Let's take a look.

PilBox Demo Apps

At the point of writing this post, there are 10 demo apps available as .zip-File: hello, steps, wecker, radio, chess, browser, battery, calc, demo and crash.

Let's take a look at "Radio" as example. You can download the zip-file from http://software-lab.de/radio.zip or do it directly from the app:

1. Touch the "settings" wheel and type "radio" into the input field of the "PILs" tab.
2. Click on "Download". A new entry called "binary" will appear in the table.
3. If you go back to the home screen (PicoLisp symbol on the upper left), you will see a button "Radio".
4. Clicking on it shows several web radios.

Reminder: Just like you shouldn't download files from unknown sources, you should also be careful what kind of zip-files you are loading into your pilbox. I'm pretty sure there is no PicoLisp malware out in the wild, but better safe than sorry :)

Functionality

Next, let's test the functionality. It is pretty simple: When we tap a station name, the radio starts playing (obviously internet connection is needed). When you tap the "stop" button at the bottom, the radio stops.

When you tap the settings wheel, you will also find a few setting options, namely "alarm", "stations" and "language".

It does what you expect - you can set an automatic alarm at a given time (format HH:MM:SS), you can change the pre-defined stations and the language.

Examine the source code

Now tap on the radio symbol until you get back to the home screen, and then click on settings. From there, we can examine the app's source code by clicking on "radio", which will list all files included in the app.

By clicking on each of the files, we can see the respective code.

Demo Apps Overview

The main purpose of the demo apps is to demonstrate how to write your own apps and access system functions. We will go further into that in one of the next posts.

Below you can find the available apps and what features they are demonstrating:

• hello.zip: A "hello world" app, which just prints a "Hello World!" headline to the screen.
• steps.zip: Needs access to the physical activity. It is a simple step counter. It gets the "steps" from the step counter built into the phone, and compares it cumulatively against a daily target.
• wecker.zip: a simple notifier. You can set date, time and title, and a automatic notification will pop up.
• radio.zip: streams web radio. You can set an alarm too.
• chess.zip: A chess game against a (not very strong) AI. You can set the difficulty by defining the "search depth".
• browser.zip: A simple web browser. You can also set bookmarks and start pages. Also, you can set the header's user-agent to arbitrary values.
• battery.zip: displays the battery status of the current device (charging rate and capacity).
• calc.zip: a bignum calculator (only integers).
• demo.zip: a collection of demo examples for typical smartphone applications: GPS, Camera, QR scan, Notification, Gaphics with canvas (the forest fire]), the Game of Life, input fields and charts.
• crash.zip: error handling example (division by 0).

Note: Unfortunately it can happen that the PilBox app terminates the process while running in the background. The screen will show "Zombie" when you re-open it. In this case, just swipe it out and restart it.

Sources

• https://picolisp.com/wiki/?pilbox
• http://software-lab.de

Installation

Like any PlayStore App, it can be installed by simply clicking on the "install" button. Opening the App shows the "Home" view of the app, which should at that point be empty.

On the upper right, there is a "Settings" icon. Clicking on it shows a page with three tabs: The first one shows a table with the installed PicoLisp apps and should be empty after installation. The second one is titled "CSS", the third one is for language settings.

On the upper left, you can find the PicoLisp Logo. Touching it toggles between the REPL and the Home screen.

As long as the App is running, the PilBox logo is displayed among the notifications. Like most Android apps, it is closed by swiping it out from the running applications view.

Analyzing the PilBox source code

Before we get deeper into how to use the app, let's quickly check out the source code. As you might guess from the description in the last post, the PilBox is quite a powerful tool, so let's make sure that we have at least a rough understanding of what the app is actually doing.

To inspect the app's source code, you can get it as APK or TGZ from here. There are several ways to analyze android app APKs. I have opened it in the "Mobile Security Framework" Tool MobSF, which allows you to do a static and dynamic analysis of the app in a quite organized way. You can install MobSF locally or access the hosted version: https://mobsf.live/.

Let's open pilBox.apk in MobSF and run the static analysis. First we see some general file and aggregated "score" information. The overall security store of 55/100 is not surprising, given that the app is very powerful (although of course it does not do anything malicious, unless you load that code by yourself).

We will see later where this rating comes from in detail.

• Permissions

Next, let's take a look at the permissions. As you might have noticed, PilBox does not require any permissions during installation.

However, depending on the app which is loaded into the box, the user could be asked for the following permissions:

GPS and activity tracking as well as microphone and camera access are rated as "dangerous" which means that explicit consent from user side is required. The "normal" permissions are granted by default.

• Android API

Next, we find an overview of the files which contain the app's soure code. By clicking on it, we could read the de-compiled code in cleartext.

• Manifest Analysis

The next analysis fields, "Browsable Activites" and "Network Security", are emtpy, but the next point "Manifest Analysis" contains some interesting information.

For one thing, it claims that the app is communicating in cleartext. This is actually not fully true: Cleartext communication is permitted, but the default communication is ssl-encrypted by the C-library libssl.so (also included in the standard pil21 distribution).

Also, application data can be backed up which means that the data can be copied unencrypted from the device when connected to a computer. Again, it depends on the stored data if this is a problem or not.

As last step, let's take a look at the shared library binaries. Here the security checker is complaining about runpath variables and missing stack canaries. I'm not 100% sure about the details, but potentially, these kind of settings can be abused by an attacker to gain system control. However, we should keep in mind that PilBox is already a powerful tool by itself, as it allows almost anything you can do within the scope of your app sandbox.

Conclusion

The PilBox is a powerful tool and allows you to do a lot of things on your smartphone which normally requires a full Android studio development environment. Obviously, this also means that the developer needs to know what they are doing. For example, for certain applications it might be needed to add additional safety features if the default ones from the PilBox app are not sufficient.

In the next post, we will see how to load and modify basic applications in to the PilBox.

Sources

• https://picolisp.com/wiki/?pilbox
• https://software-lab.de
• https://github.com/MobSF/Mobile-Security-Framework-MobSF

This series builds up on the Web App Programming and Databases series, so it might be helpful to check out those posts - or the PicoLisp documentation - first.

Note that this works for Android only, and you will need a smartphone - or emulator - with Android >= 5.0 and a Arm64 CPU (in case you're unsure: if you have a smartphone built after 2020, it is very likely that your phone fulfills the conditions). Rooting is not necessary.

This post will cover the basics of Android smartphone architecture, because I think it is helpful to understand the concept.

The Android Platform Architecture

Android is an operation system as well as a software platform for mobile devices such as smartphones, tablets and other devices. Android is free software, published by the Android Open Source Project under the Apache License. The company Google is behind Android, which is the reason why most smartphones have the Google Play Store pre-installed and require a Google account. However, depending on your needs, this it is not a must - there are alternative repositories for Android, such as F-Droid that are completely independent from Google.

The Android software stack consists of six main components: The Linux Kernel, the Hardware Abstraction Layer, Native C/C++ Libraries, the Android Runtime Environment, the Java API Framework and the System Apps, organized in five layers.

1. At its core Android has a Linux kernel and uses the key security features of Unix systems, such as a user-based permission model and process isolation.

2. On the next-higher level, we have the hardware abstraction layer (HAL) with multiple library modules. Each of the modules interface to a specific hardware component (such as camera or bluetooth). When a framework API makes a call to access device hardware, the Android system loads these modules from the library.

3. Then we find the Android Runtime (ART), which is the application runtime environment. It translates the bytecode of an application into native instructions for the processor.

4. On the same level, we have the native Android C/C++ libraries which can be directly accessed to interact with physical device input (Native Development Kit (NDK)) instead of via the Java framework APIs.

5. The "standard" access to system functions is via the Java API Framework which provides all features to build native apps, for example the view system to build the app's user interface, a resource manager, notifications- and activity managers, and content providers to access data from other apps.

6. On the top layer, we can find the Android core system apps for email, SMS, calendar, browsing, contacts and so on. The capabilities of these apps can be re-used by developers, and they can also be replaced by external applications.

The Sandbox Principle

One of the key factors for the success of smartphones was to open the devices for apps from independent developers. In other words, anyone can write and publish a smartphone app. At the same time, smartphones collect very sensitive, personal user information, such as contacts, position, microphone and camera access and so on, which makes it very attractive for hackers. One important safety feature to prevent malicious apps from gaining access to all data on the phone is the Android sandbox principle.

The Sandbox builds up on the Linux user-based protection to identify and isolate app resources. Each app is running in their own sandbox, has a unique user ID and runs its own process. By default, apps can't interact with each other and have only limited access to the OS.

Since the sandbox principle is implemented in the kernel, it is not easy to compromise it. However, it is not invulnerable. On the Android documentation page, you can see that the access control sandbox has been getting more and more restrictive which each Android version as reaction to known vulnerabilties:

Writing and Publishing Android Apps

The standard way to write Android apps is using the Android Studio and SDK, which is a rather heavy tool and includes a debugger, libraries, emulators and tutorials. The standard language to write Android Apps is in Java or Kotlin. Other languages, such as JavaScript, can be used most comfortably with the help of frameworks such as React Native or the Ionic Framework.

The finished app can then be published in repositories. The most important source for external apps is the Google Play Store. All apps hosted on the Play Store are checked for basic security features - although you can still find a lot of malware as well. Nevertheless, apps downloaded from a controlled repository such as the PlayStore guarantee a minimum safety that downloaded software has not been manipulated by malicious acters and has a minimum functionality.

The PicoLisp "PilBox"

Now finally, let's come to the point: How can we use PicoLisp to write our own apps? The principle is simple: What we need is a kind of "framework" that makes it possible to access the Java API via PicoLisp by mapping the code to Java.

For this purpose, we can use the PilBox App. It is available in the Google PlayStore and is written in Java. It displays a WebView GUI, and starts a PicoLisp binary compiled for Arm64 CPUs.

The PilBox kernel provides an interface to the Android Java runtime environment. To the PicoLisp code it looks like a remote database, where Java objects are mapped to PicoLisp DB objects, and Java functions and methods are executed via remote procedure calls.

In other words, the PilBox is just a container for our future apps. It comes with some built-in demo apps though, that we will check out in the next few posts.

One big advantage (besides that we can use PicoLisp) is that the heavy Android SDK environment can be avoided, and strictly speaking not even a laptop is required to develop the app (if you are comfortable typing on the phone).

I think this is a very interesting approach for small apps that are mainly written for your own needs. For example, one of the simple demo apps is a step counter that interfaces with the built-in step counter in the phone. Of course, you can use a pre-installed "Health"-App or download a new one, but if you don't want to share your activity with outside parties or have specific ideas how to monitor it, you could simply write your own app.

Next steps

In the next couple of weeks, I plan to cover the following topics:

• How to install the PilBox,
• How to connect from other devices (like your PC) to the PilBox for app development,
• The main components to write your own PicoLisp app,
• Modifying or expanding the existing demo apps,
• Writing a new app.

I think it will be interesting, so stay tuned :-)

Sources

• https://developer.android.com/guide/platform
• https://picolisp.com/wiki/?pilbox

The sum of the primes below 10 is 2 + 3 + 5 + 7 = 17.

Find the sum of all the primes below two million.

This task is so easy that it is almost not worth its own post. The reason is because it is identical to Task 7, "10001st prime" except that we need to add one single line.

The solution

What we need to do is:

1. Generate a list of all prime numbers below 2.000.000,
2. Sum up all list items.

For 1., we can re-use the code from Task 7 without modification. The only challenge is that instead of 10.000 prime numbers, we need to calculate around 150.000 prime numbers. Then we sum all list items up with apply:

(load "./Task_7_10001_Prime_final.l")
(prinl (apply + (sieve 2000000)))

Let's compare the computation time for Task 7 and Task 10.

: (bench (apply + (sieve 110000)))
0.114 sec
-> 544815056
: (bench (apply + (sieve 2000000)))
5.135 sec
-> 142913828922

The computation time has increased from 0.1 s to 5.1 s, but is still much faster than the 60 seconds that the task description allows.

You can find the "code" for the finished task here.

Sources

• https://projecteuler.net/problem=10