FiveForths

Devlog 39 Release V04 Toggling Leds

Mon, 23 Jan 2023 00:00:00 +0900

January 23

Log 39
Release v04
Toggling LEDs
Closing thoughts

Log 39

In this session I’ll discuss the latest release which includes handling HEX numbers and manipulating hardware.

Release v04

One of the major obstacles to working with hardware was the lack of support for hexadecimal numbers. That is, inputting something like 0xCAFE4241 would be treated as a string (and word not found). Back in the day, older Forths would set the base value using a word such as HEX or DECIMAL or even OCTAL. I think this is completely unnecessary and much more elegant to simply prefix a number with the number base 0x. It also prevents problems where one might forget to set the base back to what it previously was. Decimal is the default base for numbers, and octal can be ignored (or implemented in Forth). Binary input might be added in the future, with the 0b prefix, but I don’t think it’s very important for now.

More importantly, when browsing through the CPU datasheet, we’ll typically (always) find memory addresses given in hexadecimal values, so let’s support that.

First, in our internal number function after checking if the number is negative (prefixed with -), instead of jumping to number_digit we’ll jump to number_check and see if it’s hexadecimal (prefixed with 0x):

number_check:
    li t3, 0x00007830           # load the '0x' string into temporary
    lhu t2, 0(a0)               # load the first 2 characters into temporary
    bne t2, t3, number_digit    # jump to number digit loop if the first 2 characters are not '0x'

We’re using the lhu instruction to load a half-word (16 bits / 2 bytes), and then comparing those 2 bytes with the value 0x00007830 which just so happens to be 0x when input through the UART. If there’s no match, then we’ll jump to the number_digit routine as usual. If there is a match, then we’ll set the number base to 16 in a temporary register, and increment the buffer address by 2 to skip the 0x characters. We jump to number_error if the string is only 2 characters and equal to 0x. This doesn’t actually return an error / reset things. All it does is adjust the return value in the X working register so when the function returns, we’ll know it’s not a number (maybe 0x is a word? why not?).

In number_digit, we’re doing things a bit differently to check if the digits are valid. We want to ensure they’re between 0-9 if the base is 10, and between 0-F if the base is 16. Unfortunately in ASCII the letters A-F don’t immediately follow 9 or precede 0, so a bit of funky math is required. The exact approach is explained very well in the Forth book “Threaded Interpretive Languages (1981)”, but luckily for us it was also implemented in jonesforth. Even better, an existing QEMU jonesforth RISC-V port also implemented this approach in Assembly, so I inspired myself from those two and made some modifications to work in FiveForths.

The final change was to change the return type for the X register. Previously a 1 meant OK and a 0 meant ERROR. I changed it so 0 means OK and anything greater than that means ERROR. The reason is the X register (a1) starts off holding the size of the token, and while processing each digit we decrement that size until it reaches 0. Although if we detect a non-digit (ex: G or $) then it’s definitely not a number, so we end the routine there, leaving X at its last value, which will ultimately be greater than 0 (thus, an error). If the value of X is 0 at the end of the routine, we know for sure we have a valid number, which we store in the W working register (a0).

Of course in the interpreter when call number returns, we need to check if X was zero instead:

-    bnez a1, push_number    # push the token to the stack or memory if it's a number
+    beqz a1, push_number    # push the token to the stack or memory if it's a number

That’s a small but crucial change.

Toggling LEDs

Now that we can input hexadecimal numbers, it becomes much easier to mess with the hardware. Our Hello World for FiveForths involves toggling the blue (pin A1) and green (pin A2) LEDs.

Before we start, it’s important to define some words which will be used later:

: invert -1 nand ;
: over sp@ 4 + @ ;
: swap over over sp@ 12 + ! sp@ 4 + ! ;
: and nand invert ;
: or invert swap invert nand ;

Now here’s the Forth code to turn on the green and blue LEDs on the Longan Nano:

: green_led_on 0x40010800 @ 0xFFFFFF0F and 0x00000030 or 0x40010800 ! ;
: blue_led_on 0x40010800 @ 0xFFFFF0FF and 0x00000300 or 0x40010800 ! ;
green_led_on
blue_led_on

Here’s what the code actually does:

read the current GPIOA port config (with @)
apply a mask to clear the 4 bits for pin the (with and)
apply the new pin config (with or)
store the new GPIOA port config (with !)
execute the new word to turn on the LED

And here’s an explanation of the hexadecimal values:

0x40010800: GPIOA base address with offset 0x00 for CTL0 pins 0-7 (would be CTL1 with offset 0x04 for pins 8-15).
0xFFFFF0FF: mask to clear GPIO pin 2 (would be the same for GPIO pin 10, while GPIO pin 1 would be 0xFFFFFF0F and GPIO pin 8 would be 0xFFFFFFF0).
0x00000030: GPIO pin 1 setting 0b0011 which is push-pull output, max speed 50MHz.
0x00000040: GPIO pin 1 setting 0b0100 which is floating input.
0x00000300: GPIO pin 2 setting 0b0011 which is push-pull output, max speed 50MHz.
0x00000400: GPIO pin 2 setting 0b0100 which is floating input.

Now to turn off the LEDs is equally simple:

: green_led_off 0x40010800 @ 0xFFFFFF0F and 0x00000040 or 0x40010800 ! ;
: blue_led_off 0x40010800 @ 0xFFFFF0FF and 0x00000400 or 0x40010800 ! ;
green_led_off
blue_led_off

Notice here in the definitions all we changed is the value of the GPIO pin settings. We’re switching the GPIO pins back to floating input mode, which essentially is like turning off the LEDs (technically the LEDs are active-low, so turning them “on” requires us to bring the pins low).

I think this shows some of the stronger points of writing in Forth as opposed to Assembly, and it’s nice being able to do it interactively and see the results right away.

Closing thoughts

I’ll be focusing on other projects moving forward, so I don’t expect to add any new features or enhancements anytime soon, or at least until those projects are completed. Thanks for following along!

Devlog 38 Release V03

Thu, 19 Jan 2023 00:00:00 +0900

January 19

Log 38
Release v03
Bug fixes
Enhancements
Closing thoughts

Log 38

In this session I’ll discuss the bugs I fixed, some enhancements, and documentation for FiveForths.

Release v03

In release v0.3 I made quite a few changes, most importantly is regarding documentation, so let’s start there.

A new sub-directory was created, called docs/ which contains, you guessed it, documentation! It’s using a system called Diátaxis which was previously the Divio docs, which I believe were created by Daniele Procida. I’ve used it in previous projects over the last few years and have grown quite fond of it.

The major goal of those documents is to allow people to learn about FiveForths, to write their own Forth code and maybe even contribute some Assembly code. I’ll continue adding to it as needed, but please feel free to contribute any changes which may help (even typos in the docs).

The next important changes were some bug fixes. I opened 4 issues on GitHub and 1 pull request prior to rebase-merging into the master branch.

Bug fixes

The STORE bug in issue 14 was a real simple one. The ! (STORE) primitive should store x in addr. I’m not sure how I overlooked it, but I somehow managed to reverse the order and found myself storing addr in x. I swear I tested it, so I reviewed devlog 32 and I realize my mistake. The code:

here @ latest @ !<Enter>

It had the arguments backwards! That command would normally try to store HERE into LATEST (in a working Forth), but since it was reversed in my code (and in the example), it actually worked. It ended up storing LATEST in HERE and worked perfectly. Oops!

Here’s the fixed code in the STORE primitive:

-    lw t0, 0(sp)        # load the DSP value (x) into temporary
-    lw t1, CELL(sp)     # load the DSP value (addr) into temporary
+    lw t1, 0(sp)        # load the DSP value (addr) into temporary
+    lw t0, CELL(sp)     # load the DSP value (x) into temporary

The TOIN bug in issue 13 was a bit more complex. In fact it didn’t affect the way FiveForths worked, as it only affected the result from using >in in Forth code. With >in it should actually store the buffer position in the TIB, but in my case I was storing a memory address in TOIN, so using >in correctly would require one to subtract the value from tib in order to obtain the real >in.

I fixed this without changing too much code. The main thing I realized was that my usage of TOIN within the assembly code was fine. I just needed to make sure the value it stored and loaded to/from memory was not an address.

The first change was to initialize TOIN to zero rather than a memory address:

-    sw t0, 0(t1)        # initialize TOIN variable to contain TIB start address
+    sw zero, 0(t1)      # initialize TOIN variable to contain zero

Next, when loading TOIN, we need to add the address of TIB so the rest of the code can function as usual. Let’s have a look at the interpreter:

-    lw a1, 0(t3)                                # load TOIN address value into X working register
+    lw a1, 0(t3)                                # load TOIN value into X working register
+    add a1, a1, t2                              # add TIB to TOIN to get the start address of TOIN

Then, when storing TOIN, we need to subtract the address of TIB:

+    li t2, TIB              # load TIB memory address
     add t0, a0, a1          # add the size of the token to TOIN
+    sub t0, t0, t2          # subtract the address of TOIN from TIB to get the new size of TOIN
     sw t0, 0(t3)            # move TOIN to process the next word in the TIB

See? Here we load TIB into a temporary because it wasn’t available elsewhere, so we add the size of the token, then we subtract the value of TIB to obtain the real size of TOIN before storing it back into the variable. That’s a mouthful but it works and was super easy to implement and validate. I made the same change in COLON (which also manipulates TOIN) and we’re good to go.

Enhancements

Two of the enhancements I focused on were to ensure the user dictionary and data/return stacks don’t overflow (or underflow). We want to make sure our code stays within the memory boundaries assigned to them.

Another enhancement was regarding error handling, and I’m quite happy about this one. In sectorforth there’s practically no error handling and many Forths are somewhat cryptic about what’s going on. We want to know why something went wrong, so let’s add some proper error messages.

I noticed the code for error, ok, reboot was quite similar, and I wanted to add other types of messages, so the first step was to create a new macro to print error messages:

# print a message
.macro print_error name, size, jump
    .balign CELL
  err_\name :
    la a1, msg_\name    # load string message
    addi a2, a1, \size  # load string length
    call uart_print     # call uart print function
    j \jump             # jump when print returns
.endm

It’s fairly simple, all it does is generate a label (ex: err_error), code for loading a string message (ex: under the label msg_error), printing the message, and then jumping to a label specified as an argument.

It would be used like this:

print_error error, 4, reset
msg_error: .ascii "  ?\n"

When there’s a jump to the label err_error, it will load the ASCII error message stored in msg_error, add its size (4), print it to the UART, then jump to the reset label.

I defined similar code for other messages as well:

print_error error, 4, reset
print_error ok, 6, tib_init
print_error reboot, 16, _start
print_error tib, 14, reset
print_error mem, 16, reset
print_error token, 14, reset
print_error underflow, 20, reset
print_error overflow, 20, reset

msg_error: .ascii "  ?\n"
msg_ok: .ascii "   ok\n"
msg_reboot: .ascii "   ok rebooting\n"
msg_tib: .ascii "   ? tib full\n"
msg_mem: .ascii "  ? memory full\n"
msg_token: .ascii "  ? big token\n"
msg_underflow: .ascii "  ? stack underflow\n"
msg_overflow: .ascii "   ? stack overflow\n"

I think that’s slightly more useful than just ? and ok haha.

The new messages such as msg_mem and msg_overflow are for the code which handles the bounds checks on memory and stacks.

I won’t go into full detail of the changes required to perform bounds checks, but I will highlight a few things.

First, I defined yet another new macro, this one is for handling errors where the values stored in memory are incomplete. For example during a word lookup, if we’re in compile mode, we absolutely must not leave the memory in a half-baked state. That means restoring HERE and LATEST. The same rule applies when ; (EXIT) is encountered, if there isn’t enough memory to store that word, then the entire word’s definition must be rolled back. Here’s the new macro to handle that:

# restore HERE and LATEST variables
.macro restorevars reg
    # update HERE
    li t0, HERE         # load HERE variable into temporary
    sw \reg, 0(t0)      # store the address of LATEST back into HERE

    # update LATEST
    li t0, LATEST       # load LATEST variable into temporary
    lw t1, 0(\reg)      # load LATEST variable value into temporary
    sw t1, 0(t0)        # store LATEST word into LATEST variable
.endm

It’s quite straightforward, it simply stores the provided register value of LATEST into HERE, and then it updates LATEST to contain the previously stored word in LATEST. It sounds weird to update LATEST with LATEST but actually it’s not, it’s just storing the pointer value of LATEST into LATEST, which is a different address. Essentially it ends up restoring the old LATEST and HERE variables to the previous state.

Another important change regarding memory bounds checking… after restoring the variables and jumping to the reset handler, I absolutely want to zerofill the memory. In fact, I want to do this on every reset (physical or virtual). I don’t like the idea of leaving old values in memory after a soft reset, so some code was added to fully zero out the RAM. Of course I was careful to only reset from the HERE address, since we don’t want to lose our correctly defined words haha (if yes, then do a reboot to start from scratch):

# reset the RAM from the last defined word
ram_init:
    li t0, HERE         # load HERE memory address
    lw t0, 0(t0)        # load HERE value
    li t1, PAD          # load PAD variable
ram_zerofill:
    # initialize the memory cells
    beq t0, t1,ram_done # loop until counter (HERE) == PAD
    sw zero, 0(t0)      # zero-fill the memory address
    addi t0, t0, CELL   # increment counter by 1 CELL
    j ram_zerofill      # repeat
ram_done:
    # continue to tib_init

I guess this code is extremely similar to tib_init, where it just writes 4 bytes (zeroes) to every memory address from HERE to PAD (but it doesn’t touch PAD). I tested on the Longan Nano’s 20 KBytes and it’s pretty quick even running at 8 MHz haha. I can’t imagine this would be problematic even on a larger microcontroller, but we’ll see.. the reset only occurs on error anyways (and first boot), so it won’t affect code that’s running fine and error-free.

Finally, for bounds checks on the stacks, I modified some macros and primitives with code similar to this in the POPRSP macro:

    li t0, RSP_TOP              # load address of top of RSP
    bge s2, t0, err_underflow   # jump to error handler if stack underflow

The above code performs a stack underflow check on the return stack. All it does is jump to the err_underflow error handler if the value of the RSP pointer (s2) is equal or greater than the value of the RSP_TOP constant. Both are memory addresses, and if they’re lined up then it should not be possible to perform a POPRSP. That’s called an underflow.

For the overflow check we can look at the PUSH macro:

    li t0, RSP_TOP+CELL         # load address of bottom of stack + 1 CELL
    blt sp, t0, err_overflow    # jump to error handler if stack overflow

This is similar to POPRSP except it’s using the RSP_TOP constant + 1 CELL, which is essentially the last available memory cell in the data stack. That value is fine, but anything below it is not. That’s why we’re using the blt (branch if less than) RISC-V instruction. If there were a blte instruction (branch if less than or equal) then we could write this instead:

    li t0, RSP_TOP

There might be a simpler way, but I’ll leave that for the optimization stage (soon?).

So back to the over/under flow checks, anywhere the sp (DSP) or s2 (RSP) pointers are manipulated, there will first be a check for a stack overflow or underflow condition. If yes then we’ll print a friendly error message and jump to reset.

Closing thoughts

I believe this release is quite stable now, but there’s still a few more enhancements I want to make, such as adding the ability to save or load words either from the onboard SD card or from Flash memory. I still need to add the ability to write multi-line word definitions, and the ability to handle hex numbers. However I also want to avoid adding to much to the core Forth (and avoid writing much more low-level RISC-V Assembly).

In the next session I’ll focus on the above enhancements and maybe some code cleanup and optimization.

Devlog 37 A Release Or Two

Tue, 10 Jan 2023 00:00:00 +0900

January 10

Log 37
A release or two
Why so much work
Closing thoughts

Log 37

In this session I’ll discuss the latest changes I’ve made to the source code, and provide information on the releases.

A release or two

In the last day I’ve published 2 releases to GitHub. The first, v0.1 was a partial failure because I didn’t realize the firmware.bin did not automatically jump to the _start label. I fixed this by moving it to the src/03-interrupts.s source file:

# Ensure the _start entry label is defined first
.text
.global _start
_start:
    j boot

The backspace issue where typing it would completely mess up the TIB is now fixed. Well it turns out there was a line of code which shouldn’t have been there (so I removed it):

-    sw a1, 0(t3)            # store new TOIN value in memory

This code was storing the updated TOIN address back to memory, before we even started processing the token. Oops!

Another change was to add the lit primitive, which makes it possible to add numbers to a colon definition. For example, this works:

: add5 5 + ;
10 add5

The stack pointer would then have the decimal value 15. Here’s the code for LIT, it’s inserted right after NAND and EXIT:

# lit ( -- n )          Get the next word from IP and push it to the stack, increment IP
defcode "lit", 0x03888c4e, LIT, NAND
    lw t0, 0(s1)        # load the memory address from IP into temporary
    PUSH t0             # push the literal to the top of the stack
    addi s1, s1, CELL   # increment IP by 1 CELL
    NEXT

There’s some other code added to the push_number routine which handles inserting the LIT codeword address to a colon definition (in memory), followed by the actual number, if the STATE of the interpreter was set to 1 (compile mode).

Finally, that release saw the initial introduction of some actual documentation in the form of a README. It’s not complete and not final, as I plan to introduce proper documentation in the near future.

The second release, v0.2 is a bit more polished and has a few improvements.

The carriage return and zero control characters are now completely ignored in the interpreter function:

# ignore specific characters
mv t4, zero                                 # load 0x00 zero into temporary
beq a0, t4, interpreter                     # ignore the character if it matches
li t4, CHAR_CARRIAGE                        # load 0x0D carriage return into temporary
beq a0, t4, interpreter                     # ignore the character if it matches

We’re ignoring only those ones and not the entire set of non-printable characters because the others aren’t really problematic, but I may be wrong. It’s not a big deal either way, and can easily be changed if needed.

The location of the _start entry symbol was revised and moved to the top of the fiveforths.s source file. This guarantees it’ll be located at the start of Flash (0x08000000).

There’s now a boot message when you first boot/reset the microcontroller, the terminal will display this uneventful greeting:

FiveForths v0.2, Copyright (c) 2021~ Alexander Williams, https://a1w.ca

We now have a visual indicator that the Forth is booted and ready to accept commands. Yay!

The last three changes are mostly cosmetic but equally important:

First, I moved all these devlog posts and their related assets, html pages, build files, etc to the gh-pages git branch. The goal was to cleanup and reorganize the repository so the code remains in the master branch, separate from the devlogs website.

Next, the Makefile was heavily modified to support building different types of boards with different types of microcontrollers. Some code was split out from the src/ source files and moved into src/mcus/gd32vf103/mcu.s, and other code was moved to src/boards/longan-nano-lite/board.s. I also added a simplified linker script for each board (ex: the longan-nano has slightly more RAM and double the FLASH of the longan-nano-lite).

Now it’s possible to specify command-line variables for make to create various firmware binaries. Here are a few examples:

make build BOARD=longan-nano
make build BOARD=longan-nano-lite

There’s a few other options but they’ll be described in the upcoming documentation.

Finally, I added a GitHub Action to automate the firmware builds from this repository. At the moment it builds longan-nano and longan-nano-lite firmware, generates the sha256 hash, and uploads them as an artifact of the build process. I then use those exact files in the release. This means you don’t even need the whole RISC-V setup to try FiveForths, just grab a binary, flash it, and get to work.

Why so much work

Why put so much effort into a just for fun open source project?

Well this project was discovered by Hackaday.io and written about on Hackster.io, so I suddenly found myself with a pressing need to take this a bit more seriously. I want to make it easier for people to try it, learn more about it, and possibly even provide contributions. Did I mention it’s fun?

Closing thoughts

I want to make a random shoutout to Daniel Mangum who did a fantastic job of documenting some RISC-V things.

The next release will include the updated documentation which I believe will be very helpful for people who want a better idea about FiveForths without browsing through the source code. It will also contain some Forth code examples and other useful information.

In the next coding session, I’ll likely focus on the remaining open GitHub issues, which I labeled as “enhancements” because they aren’t critical to the functioning of FiveForths.

Devlog 36 From Dtc To Itc

Sat, 07 Jan 2023 00:00:00 +0900

January 07

Log 36
From DTC to ITC
Cleaning up lookup
Closing thoughts

Log 36

In this session I fix a few bugs which made me switch threading mode.

From DTC to ITC

In devlog 29 I mentioned my goal of implementing this Forth as a Direct Threaded Code (DTC) Forth, but I ended up banging my head on the wall trying to actually make it work correctly.

It works but it doesn’t work. The main issue was with executing compiled words which included other compiled words. The address loaded in the W register was the address of the word, not the address pointed to by the word (docol). There is likely a way to fix it, but I got annoyed with the idea that compiled words need to be executed differently from primitive words.

For that reason, I decided to model my Forth on the classic Indirect Threaded Code (ITC) approach found in many implementations such as jonesforth and derzforth.

To make these changes, we first need to modify the NEXT macro to add another level of indirection:

-    jr a0               # jump to the address in W
+    lw t0, 0(a0)        # load address from W into temporary
+    jr t0               # jump to the address in temporary

Next, I added another level of indirection for jumping to docol:

-    la a2, docol        # load the codeword address into Y working register
+    la a2, .addr        # load the codeword address into Y working register

And the .addr is defined here as a jump to docol:

+.addr:
+    j docol             # indirect jump to interpreter after executing a word

Finally, when executing a word, we want a double-indirection to the outer interpreter similar to NEXT:

-.loop: .word process_token  # indirect jump to interpreter after executing a word
+.loop: .word .dloop         # double indirect jump to interpreter
+.dloop: .word process_token # indirect jump to interpreter after executing a word

One more change I made was to the defcode macro. I wanted to have specific global labels for each part (link, hash, code), and a global label for the body which is where the Assembly code is located. This makes everything much more clear when debugging and it’s easier to trace. Here’s the full macro after modifications:

.macro defcode name, hash, label, link
    .section .rodata
    .balign CELL        # align to CELL bytes boundary
    .globl word_\label
  word_\label :
    .4byte word_\link   # 32-bit pointer to codeword of link
    .globl hash_\label
  hash_\label :
    .4byte \hash        # 32-bit hash of this word
    .globl code_\label
  code_\label :
    .4byte body_\label  # 32-bit pointer to codeword of label
    .globl body_\label
  body_\label :         # assembly code below
.endm

Now we can test some Forth code in the terminal. First we’ll define dup, then we’ll define invert, then we’ll call invert on the stack value -66, and emit that to the terminal. It should print an A which is 0x00000041 or decimal 65 (nand of -66 and -66):

: dup sp@ @ ;<Enter>   ok
: invert dup nand ;<Enter>   ok
-66 invert<Enter>   ok
emit<Enter> A   ok

Yessss!!!

Cleaning up lookup

The lookup function was not cleaning up after itself when an error was found. This was not an issue when executing words, only when compiling because it would essentially leave a word half-compiled in memory.

I think the first change is to make a copy of LATEST once we enter the function. This is the value we want to restore if there’s an error, but we only want to do it once:

 lookup:
-    beqz a1, error              # error if the address is 0 (end of the dictionary)
+    mv t2, a1                   # copy the address of LATEST

Next, we want to move our guard to the loop part, which will happen on every word that’s looked up:

+lookup_loop:
+    beqz a1, lookup_error       # error if the address is 0 (end of the dictionary)

In lookup_next, we want to jump to the loop instead, so let’s change that:

-    j lookup
+    j lookup_loop

Then we can begin to define our custom lookup error handler:

lookup_error:
    # check the STATE
    li t0, STATE                # load the address of the STATE variable into temporary
    lw t0, 0(t0)                # load the current state into a temporary
    beqz t0, error              # if in execute mode (STATE = 0), jump to error handler to reset

First want want to check the STATE of the interpreter. If we’re in execute mode then it’s safe to jump to the error function which will handle resetting things (without touching HERE or LATEST).

Otherwise, if we’re in compile mode, we want to store our previously saved LATEST value into HERE. This rolls back the memory address of HERE as if we didn’t even define a word:

    # update HERE since we're in compile mode
    li t0, HERE                 # load HERE variable into temporary
    sw t2, 0(t0)                # store the address of LATEST back into HERE

Next, we want to update LATEST so it points back to the previous word that was defined before the current one. That address is actually still there in memory, at the location pointed to by HERE (the t2 register from earlier):

    # update LATEST since we're in compile mode
    li t0, LATEST               # load LATEST variable into temporary
    lw t1, 0(t2)                # load LATEST variable value into temporary
    sw t1, 0(t0)                # store LATEST word into LATEST variable

Once that’s done, we can jump to the error function to handle resetting other things:

    j error                     # jump to error handler

Closing thoughts

Alright, now everything actually works!! (I hope)

Now there’s only one small bug remaining, which is related to hitting backspace in the terminal. It’s probably a small issue, but I’ll get to that eventually. In the next session, I’ll publish the README, and work on the documentation, examples, optimizations, and code cleanup.

Devlog 35 Compiling Words Pt2

Fri, 06 Jan 2023 00:00:00 +0900

January 06

Log 35
Compiling words pt2
Done
Closing thoughts

Log 35

In this session, I actually plan on fixing compilation and getting to DONE!

Compiling words pt2

I decided to step through the execution of a compiled word: dup using GDB.

The first problem I noticed was the indirect jump to docol in COLON was not working. In fact, it doesn’t need any indirection since we’re actually jumping straight to it. Let’s fix that:

-    la a2, .addr        # load the codeword address into Y working register
+    la a2, docol        # load the codeword address into Y working register

And then we can get rid of .addr:

-.addr: .word docol      # indirect jump to docol from a colon definition

Next issue was in docol, we don’t expect the code field address in Y, that makes no sense. We expect it in W, like in every other Forth:

-    addi s1, a2, CELL   # skip code field in Y by adding a CELL, store it in IP
+    addi s1, a0, CELL   # skip code field in W by adding 1 CELL, store it in IP

Finally, the macro defcode for defining a word was completely wacky. It had a mix of code from sectorforth and jonesforth and some weirdness added by me because I store a hash of the word instead of the length+name. In any case, I had to rewrite the entire macro and I ended up with this:

.macro defcode name, hash, label, link
    .section .rodata
    .balign CELL        # align to CELL bytes boundary
    .globl word_\label
  word_\label :
    .4byte word_\link   # 32-bit pointer to codeword of link
    .globl hash_\label
  hash_\label :
    .4byte \hash        # 32-bit hash of this word
    .4byte code_\label  # 32-bit pointer to codeword of label
    .globl code_\label
  code_\label :         # assembly code below
.endm

This is a bit better. The codefield now points to hash_\label+4, which will move directly to the code_\label. I think that adds an extra cycle and would rather have the code jump to code_\label, but when I do that the interpreter crashes… I guess I’ll need to fix that another time.

For the time being, let’s test out our dup once more in the terminal:

: dup sp@ @ ;<Enter>  ok

And let’s inspect the 6 values in GDB:

(gdb) x/6xw 0x20000000
0x20000000:	0x080004d0	0x03886bce	0x080004c0	0x080002b4
0x20000010:	0x0800027c	0x08000344

The first 2 are the link to the previous word and the hash.. unchanged since the previous devlog:

(gdb) x/xw 0x080004c0
0x80004c0 <docol>:	0xfe992e23
(gdb) x/xw 0x080002b4
0x80002b4 <hash_DSPFETCH+4>:	0x080002b8
(gdb) x/xw 0x0800027c
0x800027c <hash_FETCH+4>:	0x08000280
(gdb) x/xw 0x08000344
0x8000344 <code_EXIT>:	0x00092483

Perfect!! (almost).

If I gather up the courage to fix the issue I mentioned above, it would look like docol, code_DSPFETCH, code_FETCH, code_EXIT. Now let’s try running dup in the terminal:

123 dup<Enter>  ok

This should leave 123 as the first two entries in the stack. Let’s check the stack with GDB:

(gdb) i r sp
sp             0x20004ff8 0x20004ff8
(gdb) x/dw 0x20004ff8
0x20004ff8: 123
(gdb) x/dw 0x20004ff8+4
0x20004ffc: 123

Great!

Now we have confirmation that we can execute AND compile words!!

Done

And there we have it, my first fully functional Forth (and programming language written from scratch).

Closing thoughts

This is super exciting! There are still a few minor bugs to fix and features to add, but I’ll focus on bugs/optimizations first, code cleanup and comments, and maybe getting some examples and a README up for others to use this.

Devlog 34 Compiling Words

Fri, 06 Jan 2023 00:00:00 +0900

January 06

Log 34
Printing strings
Rebooting the MCU
Compiling words
Closing thoughts

Log 34

In this session I will work on compiling words, but before that I want to add 2 new features to my Forth: printing strings and rebooting the mcu.

Printing strings

The previous approach to printing strings over the UART was to repeat the same 2 lines of code for every character. To print ` ok\n` we would write this:

 ok:
    li a0, CHAR_SPACE
    call uart_put
    li a0, 'o'
    call uart_put
    li a0, 'k'
    call uart_put
    li a0, CHAR_NEWLINE
    call uart_put

This works fine for short strings, but it’s quite bothersome and ugly for longer strings.

Here’s my short implementation of a UART “print” function:

uart_print:
    mv s3, ra                   # save the return address
uart_print_loop:
    beq a1, a2, uart_print_done # done if we've printed all characters
    lbu a0, 0(a1)               # load 1 character from the message string
    call uart_put
    addi a1, a1, 1              # increment the address by 1
    j uart_print_loop           # loop to print the next message
uart_print_done:
    mv ra, s3                   # restore the return address
    ret

It accepts 2 arguments:

a1 which contains the memory address of the start of a string (I’ll show an example later).
a2 which contains the address of the string + its length.

The uart_print starts by saving the ra register and ends by restoring it. We do this because the call uart_put would otherwise clobber ra and it would be unable to return after printing.

The uart_print_loop simply loops over each character in the string, printing a character at each iteration. It increments the string’s address (a1) until it’s the same as a2.

Here’s how we would use it instead of the above ok function:

ok:
    la a1, msg_ok       # load string message
    addi a2, a1, 6      # load string length
    call uart_print     # call uart print function

And we could define the msg_ok string like this:

msg_ok: .ascii "   ok\n"

Note the .ascii string is NOT null terminated, and it must be aligned to 2 bytes. In other words a 3 or 5 byte string would not work.

Let’s do something similar to error:

error:
    la a1, msg_error    # load string message
    addi a2, a1, 4      # load string length
    call uart_print     # call uart print function

And define the msg_error like this:

msg_error: .ascii "  ?\n"

Rebooting the MCU

I often find myself wanting to test a clean slate of the Forth, without physically resetting the device (which requires restarting openocd and gdb). So I decided to add a new primitive called reboot, which jumps directly to the _start initialization procedure:

# reboot ( -- )         # Reboot the entire system and initialize memory
defcode "reboot", 0x06266b70, REBOOT, NULL
    j reboot            # jump to reboot

I then had to modify FETCH to link to REBOOT instead of NULL:

-defcode "@", 0x0102b5e5, FETCH, NULL
+defcode "@", 0x0102b5e5, FETCH, REBOOT

Now let’s define the reboot function:

reboot:
    la a1, msg_reboot   # load string message
    addi a2, a1, 12     # load string length
    call uart_print     # call uart print function
    j _start            # reboot when print returns

It’s pretty much the same as ok and error, with a different string message and different jump to address. Here’s the message:

msg_reboot: .ascii "  rebooting\n"

So now typing reboot<Enter> in the terminal will display the string rebooting and everything will be reset as if we first booted the device. Of course I realize this might be problematic once interrupts are enabled, but I think by then I’ll be able to remove this primitive and functionality.

Compiling words

Now the final missing element of this Forth, compiling words!!

The first change is to fix some minor issues in our macros. In 3 macros we’re decrementing the sp stack pointer by 1 CELL before performing an operation, which is fine except when that operation involves the sp pointer. Let’s change the PUSH macro first, and I’ll explain the difference afterwards:

 .macro PUSH reg
+    sw \reg, -CELL(sp)  # store the value in the register to the top of the DSP
     addi sp, sp, -CELL  # move the DSP down by 1 cell
-    sw \reg, 0(sp)      # store the value in the register to the top of the DSP
 .endm

Here we moved the sw instruction so it’s performed first, before decrementing the pointer. But we’re also storing it at the -4 offset. This was necessary for something like PUSH sp to work, where we want to push the current sp address not the next address (sp - 4).

We’ll make a similar change to PUSHRSP:

 .macro PUSHRSP reg
+    sw \reg, -CELL(s2)  # store value from register into RSP
     addi s2, s2, -CELL  # decrement RSP by 1 cell
-    sw \reg, 0(s2)      # store value from register into RSP
 .endm

And finally we’ll also modify PUSHVAR to load the register and then store it in sp - 4 before moving the sp pointer down by 1 CELL.

 .macro PUSHVAR var
-    addi sp, sp, -CELL  # move the DSP down by 1 cell
     li t0, \var         # load variable into temporary
-    sw t0, 0(sp)        # store the variable value to the top of the DSP
+    sw t0, -CELL(sp)    # store the variable value to the top of the DSP
+    addi sp, sp, -CELL  # move the DSP down by 1 cell
 .endm

In the COLON primitive (inner interpreter), we need to do the exact same thing as in process_token (outer interpreter) before and after calling token, so let’s replace the existing code:

 defcode ":", 0x0102b5df, COLON, LATEST
-    li a0, TIB          # load TIB into W
-    li t3, TOIN         # load the TOIN variable into unused temporary register
-    lw a1, 0(t3)        # load TOIN address value into X working register
+    li t3, TOIN         # load TOIN variable into unused temporary register
+    lw a0, 0(t3)        # load TOIN address value into temporary
     call token          # read the token

+    # move TOIN
+    add t0, a0, a1      # add the size of the token to TOIN
+    sw t0, 0(t3)        # move TOIN to process the next word in the TIB
+
     # bounds checks on token size
-    beqz a1, error      # error if token size is 0
+    beqz a1, ok         # ok if token size is 0
     li t0, 32           # load max token size  (2^5 = 32) in temporary
     bgtu a1, t0, error  # error if token size is greater than 32

-    # store the word then hash it
-    sw a0, 0(t3)        # store new address into TOIN variable
     call djb2_hash      # hash the token

Now COLON’s first few lines are identical to process_token.

We’ll also need to fix a bug I discovered when storing the code_EXIT address at the end of a word:

-    sw t1, 0(t0)        # store the codeword address into HERE
+    sw t1, 0(t2)        # store the codeword address into HERE

The actual HERE address was stored in t2 but I accidentally used t0 which means EXIT would not be written to the correct memory location.

Now let’s look at our compile function, called from the process_token (outer interpreter). The first step is to find the codeword address, which is 2 CELLs down:

compile:
    addi t0, a1, 2*CELL     # increment the address of the found word by 8 to get the codeword address

Then we’ll load HERE into a temporary, and store the codeword in there:

    li t1, HERE             # load HERE variable into temporary
    lw t2, 0(t1)            # load HERE value into temporary
    sw t0, 0(t2)            # write the address of the codeword to the current definition

Afterwards we can increment HERE by 1 CELL and store its value back, before jumping back to process the next token:

    addi t0, t2, CELL       # increment HERE by 4
    sw t0, 0(t1)            # store new HERE address
compile_done:
    j process_token

That’s it!

At least.. I think that’s it. Let’s try to compile a word in the terminal:

: dup sp@ @ ;<Enter>  ok

So far so good, maybe? Let’s check the user dictionary with GDB. This should store 6 values in memory starting from 0x20000000, three values for dup (link, hash, codeword), one address for sp@ (DSPFETCH), one address for @ (FETCH) and one address for exit (code_EXIT):

(gdb) x/6xw 0x20000000
0x20000000:	0x08000650	0x03886bce	0x080003f4	0x080005b0
0x20000010:	0x08000598	0x080005ec

Now let’s look at each value:

(gdb) x/xw 0x08000650
0x8000650 <word_SEMI>:	0x08000644

That’s our link to the previous word. Then 0x03886bce is the hash of the word dup.

(gdb) x/xw 0x080003f4
0x80003f4 <.addr>:	0x080003e4
(gdb) x/xw 0x080003e4
0x80003e4 <docol>:	0xfe992e23

Next we have the address of .addr which points to docol. This is where I’m still a bit confused, and it might be totally wrong.

Next let’s examine the remaining 3 values:

(gdb) x/xw 0x080005b0
0x80005b0 <code_DSPFETCH>:	0x08000284
(gdb) x/xw 0x08000598
0x8000598 <code_FETCH>:	0x08000264
(gdb) x/xw 0x080005ec
0x80005ec <code_EXIT>:	0x080002d4

All that looks pretty good to me. Let’s store a value in the stack, and then use dup to duplicate it on the stack (which is what sp@ @ does):

456 dup<Enter>

…crash

Well… I guess that doesn’t work. The word was definitely compiled and stored in memory, but there’s clearly something wrong in there. I have a feeling this might be related to the compiled .addr -> docol address, but I’m not sure.

Closing thoughts

In the next session I’ll manually step through the execution of my newly defined dup word, and see if I can find the problem. Hopefully I’ll be able to fix this in the next session, and then I’ll have a fully functional Forth. Yay!

Devlog 33 Storing Numbers

Thu, 05 Jan 2023 00:00:00 +0900

January 05, 2023

Log 33
Storing numbers
Closing thoughts

Log 33

I know I promised I would get to compile words, but that’s hard so instead in this session i’ll add the ability to store numbers on the stack. Once that works then I’ll get to compiling words hahaha.

Storing numbers

The number routine has already been written in src/05-internal-functions.s, but we had no code to call it. Unlike most Forths, I don’t want to check if it’s a number after checking if it’s a known word. That concept seems strange to me. For starters, we already know what a number will look like. Since we’re only dealing with base 10 numbers for the moment, we can define a number as “a series of digits optionally prefixed by a minus sign”.

In that case, I’d rather we scan for a number before calling djb2_hash.

One oversight is the registers need to be saved before calling number, because it clobbers the a0 and a1 registers. I could fix this but I’ll save that for the optimization step. For now, let’s just save/restore them:

    # check if the token is a number
    mv t5, a0               # save a0 temporarily
    mv t6, a1               # save a1 temporarily
    call number             # try to convert the token to an integer
    bnez a1, push_number    # push the token to the stack if it's a number
    mv a0, t5               # restore a0
    mv a1, t6               # restore a1

Here we’re saving the W and X working registers to temporary registers, and then calling number, which will return the result of the operation in X. If the result is not 0 (ex: 1), then we have a valid number so we’ll call push_number to store the number (stored in W) on the stack:

push_number:
    PUSH a0                 # push the W working register to the top of the data stack
    j process_token         # jump back to process the next token

This then jumps right back to processing the next token, completely skipping the hash/lookup/compile/execute steps. Let’s try adding numbers and a variable to the stack in the terminal:

-2147483648 -2147483649 state 4294967296 -100 100 12345 -31234567<Enter>

A few notes: -2147483649 should translate to 2147483647 as mentioned in devlog 26. The state variable will be its address, which is 0x20004cfc (or decimal 536890620), and 4294967296 should translate to 0. Let’s check in GDB, starting at the top of the stack and going down by 32 bytes (4 bytes x 8 values):

(gdb) x/8dw 0x20005000-32
0x20004fe0:	-31234567	12345	100	-100
0x20004ff0:	0	536890620	2147483647	-2147483648

Great! That works just as expected. Now we can store numbers on the stack, and non-numbers will be hashed and searched for in the dictionary.

Closing thoughts

Well that was easier than expected, let’s just hope I didn’t make a fatal mistake.. but so far it seems to work fine. In the next session I have no other choice but to jump to the compile mode and try to get that working. It might require a review of COLON and SEMI… we’ll see.

Devlog 32 Fixing Bugs

Wed, 04 Jan 2023 00:00:00 +0900

January 04, 2023

Log 32
Fixing DSP
Fixing macros
Fixing primitives
Closing thoughts

Log 32

In this session I’ll focus on fixing some bugs I discovered with the data stack.

Fixing DSP

In devlog 2, I introduced the idea of a TOS (top of stack) register. It’s not new to Forth, but it was new to me. The idea was to use a saved register (s3) to keep the top element of the stack. It would slightly simplify certain operations but also complicate others.

One of the biggest issues I’ve run into is regarding the initialization routine, particularly this line:

    mv s3, zero                 # initialize TOS register

This sets the TOS register to a known value: 0. So far so good, however let’s take a look at our PUSH macro:

# push register to top of stack and move TOS to DSP
.macro PUSH reg
    addi sp, sp, -CELL  # move the DSP down by 1 cell
    sw s3, 0(sp)        # store the value in the TOS to the top of the DSP
    mv s3, \reg         # copy register into TOS
.endm

It first decrements the DSP (sp) pointer by 1 CELL before storing the TOS value into it, and then copying whatever register was pushed.

Here’s an example:

PUSH a0

Seems simple, but it’s bad. The very first PUSH will actually end up copying the value 0 to the top of the DSP (pointed at by sp, which will be address 0x20004FFC), and it will store the value from a0 into TOS (s3). So now, our stack actually has 2 elements! Oops!! In fact, the very first PUSH should only write to TOS without moving the stack pointer… thus ignoring the 0 in the TOS, but I think coding for that condition is a bit ridiculous because the TOS could potentially have a value we want, ex: if 0= was the first command, then TOS would contain 1…

So, at this point I just want to get rid of the TOS (s3) register and only use the DSP. The entire time I’ve always had to think extra hard about the TOS register and now that I’ve encountered this bug, I just want it to disappear.

Fixing macros

To start, I’ll edit the macros in src/02-macros.s to only use the DSP. Here are the changes to the POP macro:

 .macro POP reg
-    mv \reg, s3         # copy TOS to register
-    lw s3, 0(sp)        # load DSP value to register
+    lw \reg, 0(sp)      # load DSP value to register
     addi sp, sp, CELL   # move the DSP up by 1 cell
 .endm

and the PUSH macro:

 .macro PUSH reg
     addi sp, sp, -CELL  # move the DSP down by 1 cell
-    sw s3, 0(sp)        # store the value in the TOS to the top of the DSP
-    mv s3, \reg         # copy register into TOS
+    sw \reg, 0(sp)      # store the value in the register to the top of the DSP
 .endm

The above macros were simplified thanks to the removal of the TOS. Similarly, we’ll also adjust the PUSHVAR macro (which I think I coded incorrectly anyways):

 .macro PUSHVAR var
     addi sp, sp, -CELL  # move the DSP down by 1 cell
-    sw s3, 0(sp)        # store the value in the TOS to the top of the DSP
     li t0, \var         # load variable into temporary
-    lw s3, 0(t0)        # load variable address value into TOS
+    sw t0, 0(sp)        # store the variable value to the top of the DSP
 .endm

Here we’re not loading the value pointed at by the variable anymore. Instead we’re just storing the memory address of the variable to the top of the stack. I guess we can then use @ to fetch the actual contents of those memory addresses.

Fixing primitives

That conveniently leads us to our next changes in src/08-forth-primitives.s, where we’ll start by modifying FETCH:

 defcode "@", 0x0102b5e5, FETCH, NULL
-    lw s3, 0(s3)        # load address value from TOS (addr) into TOS (x)
+    lw t0, 0(sp)        # load the top of stack into temporary
+    lw t0, 0(t0)        # load the value from the temporary (addr)
+    sw t0, 0(sp)        # store the value back the top of stack (x)
     NEXT

Yikes! We’ve got way more instructions for this, because now we need to load the value in the stack pointer, get the address it points to, then store that back into the stack pointer. Slightly more complicated than simply loading from TOS into TOS.

We can test that it works in the terminal with:

latest @<Enter>

Now if everything worked well, we should have the memory address of word_SEMI stored as the first entry in the data stack. Let’s check with GDB:

(gdb) i r sp
sp             0x20004ffc	0x20004ffc
(gdb) x/xw 0x20004ffc
0x20004ffc:	0x080004d8
(gdb) x/xw 0x080004d8
0x80004d8 <word_SEMI>:	0x080004cc

Great!

Next we’ll modify STORE by loading the top two stack entries into temporaries, and then storing one into the other:

 defcode "!", 0x0102b5c6, STORE, FETCH
     lw t0, 0(sp)        # load the DSP value (x) into temporary
-    sw t0, 0(s3)        # store temporary into address stored in TOS (addr)
-    lw s3, CELL(sp)     # load second value in DSP to TOS
+    lw t1, CELL(sp)     # load the DSP value (addr) into temporary
+    sw t0, 0(t1)        # store x into addr
     addi sp, sp, 2*CELL # move DSP up by 2 cells
     NEXT

The instruction count remains the same, but we’re not messing with TOS anymore. Let’s test it out by trying to store the value of latest to the writeable memory address HERE, which is set to 0x20000000 (the start of the dictionary) on initialization. In the terminal we’ll type:

here @ latest @ !<Enter>

Still following along? We just put HERE in the stack, and then put LATEST in the stack. Then we called STORE which technically should store 0x080004d8 into 0x20000000. Let’s check with GDB:

(gdb) x/xw 0x20000000
0x20000000:	0x080004d8

Sweet!

Actually, at this point I’m starting to feel amazing. So far everything is working as I hoped. Let’s continue with ZEQU, which was a buggy non-sensical little 1-liner:

 defcode "0=", 0x025970b2, ZEQU, RSPFETCH
-    seqz s3, s3         # store 1 in TOS if TOS is equal to 0, otherwise store 0
+    lw t0, 0(sp)        # load the DSP value (x) into temporary
+    snez t0, t0         # store 0 in temporary if it's equal to 0, otherwise store 1
+    addi t0, t0, -1     # store -1 in temporary if it's 0, otherwise store 0
+    sw t0, 0(sp)        # store value back into the top of the stack
     NEXT

First I realized the ZEQU should actually store -1 not 1. In this case I’m using the exact same approach as sectorforth but with RISC-V instructions to “set if not equal” and then to decrement the value by 1. Let’s test it by storing the STATE (should be 0) in the DSP, and then calling 0= in the terminal with:

state @ 0=<Enter>

In GDB we should find -1 as the top stack value:

(gdb) i r sp
sp             0x20004ffc	0x20004ffc
(gdb) x/dw 0x20004ffc
0x20004ffc:	-1

Now let’s call 0= again in the terminal:

0=<Enter>

And let’s inspect it again in GDB:

(gdb) i r sp
sp             0x20004ffc	0x20004ffc
(gdb) x/dw 0x20004ffc
0x20004ffc:	0

Awesome!

The last primitives to fix are ADD and NAND which should be fairly similar as they have the same stack effects. Let’s start with ADD:

 defcode "+", 0x0102b5d0, ADD, ZEQU
     POP t0              # pop value into temporary
-    add s3, s3, t0      # add values and store in TOS
+    lw t1, 0(sp)        # load DSP value (x2) into temporary
+    add t0, t0, t1      # add the two values
+    sw t0, 0(sp)        # store the value into the top of the stack

Again, without the TOS it’s a bit more complex. Here we’re still popping the top value from the stack into t0, but we’re also loading the next top value into t1 (note: the POP t0 moves the stack pointer, so we’re still loading from offset 0). Afterwards we’re adding the two registers and then storing the result back into the top of the stack.

Now let’s look at the bitwise NAND:

 defcode "nand", 0x049b0c66, NAND, ADD
     POP t0              # pop value into temporary
-    and s3, s3, t0      # store bitwise AND of temporary and TOS into TOS
-    not s3, s3          # store bitwise NOT of TOS into TOS
+    lw t1, 0(sp)        # load DSP value (x2) into temporary
+    and t0, t0, t1      # perform bitwise AND of the two values
+    not t0, t0          # perform bitwise NOT of the value
+    sw t0, 0(sp)        # store the value into the top of the stack
     NEXT

It’s almost identical to ADD except we’re performing bitwise AND and NOT instead of ADD.

That completes our changes to the forth primitives. Let’s test ADD and NAND. We’ll start by trying to add the value of LATEST to the value of LATEST. In the terminal we’ll type:

latest @ latest @ +<Enter>

Now in GDB, let’s get the address of LATEST (aka word_SEMI), right now it gives us 0x80004e8 (because we added some new instructions previously). Multiplying it by 2 should give us 0x100009D0 stored at the top of the stack.

Let’s check:

(gdb) x/xw word_SEMI
0x80004e8 <word_SEMI>:	0x080004dc
(gdb) i r sp
sp             0x20004ffc	0x20004ffc
(gdb) x/xw 0x20004ffc
0x20004ffc:	0x100009d0

Perfect!

Next we’ll see if NAND works as expected by storing STATE in the top of the stack. In the terminal we’ll type:

state @ state @ nand<Enter>

And in GBD, since we know that STATE is set to 0 when we’re in execute mode, performing a NAND of 0 and 0 should give us -1 (remember this is bitwise, which flips all the 0 bits to 1, giving us 0xFFFFFFFF or -1):

(gdb) i r sp
sp             0x20004ffc	0x20004ffc
(gdb) x/dw 0x20004ffc
0x20004ffc:	-1

Yesss!!!

That’s all for our forth primitives. The one final change is to remove the initialization of s3 in src/06-initialization.s:

-    mv s3, zero                 # initialize TOS register

Closing thoughts

This was a rather long session of writing and testing and editing code, but we did it! Not only did I fix some bugs in some of the primitives, but I also greatly simplified the data stack by getting rid of the TOS.

In the next session, I’ll get the compiler working so I can finally add words to the user dictionary starting at address 0x20000000 (on the longan nano).

Devlog 31 Interpreting Words

Wed, 04 Jan 2023 00:00:00 +0900

January 04, 2023

Log 31
Reviewing token
Interpreting words
Closing thoughts

Log 31

In this session I’m going to review the token function from src/05-internal-functions.s before reviewing the interpreter in file src/09-interpreter.s.

Reviewing token

In my token function, I read characters backward from TOIN to the start of the word. This makes no sense because it’ll then be impossible to read the next word. I’ll need to make some adjustments for the TOIN variable location, but let’s start by fixing token.

First, since we’re not comparing TOIN with the TIB address anymore, we can remove that guard:

-    blt a1, a0, token_done      # compare the address of TOIN with the address of TIB

Next, we want to increment TOIN by 1 for each character found, instead of decrementing:

-    addi a1, a1, -1             # move TOIN pointer down
+    addi a0, a0, 1              # move buffer pointer up
+    beqz t0, token_zero         # compare char with 0

I moved the buffer address to W instead of X. Here I also added a new branch address called token_zero. The TIB will always be filled with zeroes first, so we want to keep scanning the TIB until we find a zero. That’s a character which can’t be input via the UART/serial terminal, and doesn’t belong in the terminal input buffer (except when we initialize it).

Next, when we find a space character, we want to ignore it because we know we’re done scanning our token:

+    addi a0, a0, -1             # move buffer pointer down to ignore the space character
+    sub a0, a0, t2              # store the start address in W
+    j token_done

We’ll use that opportunity to store the new TOIN address in the W (a0) working register.

Our token_zero label looks like this:

+token_zero:
+    addi a0, a0, -1             # move buffer pointer down to ignore the 0

It does this because we also want to ignore the zero if we find it (by moving TOIN back by 1). This will automatically fall into token_done afterwards, whose only job is to store the size of the token in the X (a1) working register, and then return from the function:

token_done:
    mv a1, t2                   # store the size in X
    ret

That’s all for adjusting the token function. Now we can jump back to the interpreter and fix that.

Interpreting words

The first change I want to make is ensure our .loop function jumps to process_token instead of ok. We obviously don’t want to print ok after scanning only one word in the buffer:

-.loop: .word ok             # indirect jump to interpreter after executing a word
+.loop: .word process_token  # indirect jump to interpreter after executing a word

Next, in our interpreter_tib function, we don’t actually want to start processing the token just yet, first we need to replace the newline with a space, so let’s jump to a different label:

-    beq a0, t0, process_token                   # process the token if it matches
+    beq a0, t0, replace_newline                 # process the token if it matches

Then, where we convert the newline to a space, we’ll change the label name as well:

-process_token:
+replace_newline:

Now things start to make a bit more sense, let’s re-add the process_token label a bit lower, where we actually “process” the “token” right before calling the token function. This is where we’ll loop to after executing a word:

+process_token:
     # process the token
-    mv a0, t2               # load the TIB address in the W working register
+    li t3, TOIN             # load TOIN variable into unused temporary register
+    lw a0, 0(t3)            # load TOIN address value into temporary
     call token              # read the token

We made a few changes here, the first was removing the naive approach of just copying the TIB address into W.

This can probably be optimized by renaming some registers, but for now we want to reload TOIN into a temporary, and then load the TOIN address into W, before calling token. This ensures it’ll get the correct buffer address to start with.

Once the call returns, we won’t need to reload TOIN because we’ll already have it in W (a0), as well as the token’s size in X (a1). We can just add them together to get the new TOIN address:

-    lw t0, 0(t3)            # load TOIN address value into temporary
-    add t0, t0, a1          # add the size of the token to TOIN
+    add t0, a0, a1          # add the size of the token to TOIN
     sw t0, 0(t3)            # move TOIN to process the next word in the TIB

     # bounds checks on token size
     beqz a1, ok             # ok if token size is 0

This seems to work well. In the event where our token size is 0, because the last character was a space or a 0, then we’re done processing tokens and the bounds check will jump to ok, which re-initializes the TIB before looping back to the interpreter. Note the stack is not reset, so our sp stack pointer will remain and the words we entered in the terminal will have executed and their results will be stored in (or removed from) the stack (assuming they actually modify the stack).

Example:

(gdb) i r sp
sp             0x20004ffc	0x20004ffc
(gdb) x/xw 0x20004ffc
0x20004ffc:	0x00000000
(gdb) c
Continuing.

Breakpoint 1, interpreter_start () at src/09-interpreter.s:11

Now in the terminal I type latest<Enter:

latest ok

And back in GDB:

(gdb) i r sp
sp             0x20004ff8	0x20004ff8
(gdb) x/xw 0x20004ff8
0x20004ff8:	0x080004f8
(gdb) x/xw 0x080004f8
0x80004f8 <word_SEMI>:	0x080004ec

Perfect! There’s the address of the latest word we defined: SEMI.

Closing thoughts

I’m happy I can now execute multiple words from the terminal. The next step is to try and compile words. I’ll also need to ensure we call number so we can store numbers in the stack. I’ll try to focus on those two things in the next session, and see if I can find any bugs in the current implementation. Almost done!

Devlog 30 Cleanup

Mon, 02 Jan 2023 00:00:00 +0900

January 02, 2023

Log 30
Cleanup
Closing thoughts

Log 30

I can’t seem to find my way through fiveforths.s anymore. The file has grown to over 700 lines and even with a good text editor I have trouble finding stuff. In this session I’ll split things into various files to simplify my life.

Cleanup

Since GNU Assembler has very primitive include functionality, I’ll make sure to move the source files to the src/ sub-directory, and each file will be prefixed with a number, to clearly indicate the correct loading order.

The first change is to include src in the search path for the assembler, so we’ll modify the Makefile like this:

-               $(AS) $(CFLAGS) -march=$(ARCH) -o $@ $<
+               $(AS) $(CFLAGS) -march=$(ARCH) -I src -o $@ $<

Next, I modified fiveforths.s by including each file in numerical order:

.include "01-variables-constants.s"
.include "02-macros.s"
.include "03-interrupts.s"

# include board-specific functions
.include "gd32vf103.s"

.include "04-io-helpers.s"
.include "05-internal-functions.s"
.include "06-initialization.s"
.include "07-error-handling.s"
.include "08-forth-primitives.s"
.include "09-interpreter.s"

Finally, I simply moved the pieces of code into separate files. It seems like quite a big change, but the functionality remains exactly the same. Just type make and it will rebuild everything as usual.

Closing thoughts

This change was purely aesthetic but I think it was really necessary to preserve my sanity. I’ll get back to fixing the execute issue in the next session.