SD Card Access for a Arty A7: Part 7

Foreword

In the previous post we managed to read a sector of data from an SD Card.

In this post w will continue our journey and see if we can read a file from a FAT32 formatted partition. To be able to read a file from a partition will form an important part of being able to boot an Amiga core on a Arty A7, and thus to load a Boot ROM and disk image from SD Card.

There is quite number of technical details involved to read a file from a FAT32 partition. Writing the functionality for this right from the start in 6502 Assembly language is quite a daunting task.

To make our lives easier we will start to write the functionality in a High Level language. As my current knowledge of FAT32 is rather limited, experimenting with such a partition in a High Level Language will get one quickly up to speed.

Once we know how to read a file from a FAT32 partition, we can write 6502 assembly for this in a future post.

About MBR and FAT32

Before we look in detail how to read a file from an SD Card, let us start with some basic terminology.

Firstly, the storage of any SD Card is divided into many blocks, where each block is 512 bytes in size. The choice of 512 bytes per block is actually rooted in the history of Personal computers where almost any Floppy Disk Drive or Hard Drive had a basic block size of 512 bytes.

To address blocks, all the blocks are numbered consecutively starting at block 0, and going up to the maximum number of blocks the device supports.

Block zero is called the MBR or master boot record. This is also where the history of the IBM PC kicks in again and I think is probably still relevant today. When an IBM PC boots up, it looks for machine language program at block zero, to start the booting of the system. Hence the name Master Boot Record.

The MBR has some other purpose as well, which is to store one or more partition entries, so you can be able to create more than one volume on the same device. We briefly encountered this in the previous post where we saw data at location 0x1be of block 0. This data was in actual fact a partition entry.

Also, from the previous post, you will remember that apart from the partition entry data, all the other bytes of that block were zero. So, although we call block zero the MBR, the SD Cards you use today most probably will not contain machine code in that block.

Now back to the partition entry. A partition entry gives us the block number of the sector of the partition in question. This first sector of the partition is called again, surprise, a boot record! The name is again because of the legacy of the IBM PC.

The boot record also contain some data to hold of the File allocation table and to be able to read the contents of a file.

We will have a look at a typical boot record in the next section.

Looking at the boot record

To look at the boot record, we first need find the block number of the it, via the MBR. In the previous post I made a screenshot of the MBR from the SD Card I was playing with, which contained the partition entry.

Here is the screenshot again, but with the bytes of the partition entry highlighted:

As you can see the partition entry starts at 0x1be and is 16 bytes in size. To get the meaning of the bytes, we look at the following link:

https://en.wikipedia.org/wiki/Master_boot_record#PTE

Two pieces of useful information for us, in the last 8 bytes of the entry:

• offset 8: LBA of first absolute sector in the partition (4 bytes)
• offset C: Number of sectors in partition.

For interest sake, let us start by checking if we can calculate the size of the partition. The bytes at offset set are as follows: EB 0B 76 00

These bytes are in little endian order, so we need to reverse them: 00 76 0B EB. This gives us the number of sectors in hexadecimal. In decimal this is 7,736,299. To verify if this number is correct, we need to convert number of sectors to bytes, that is multiplying by 512:

3960,985,088

With the thousand separators, we get more or less to 4GB, which is the size of the SD Card I am using.

Next, let us determine the staring block of the FAT32 partition. The data bytes for this 15 20 00 00. Swop this around because of little endianness:

00 00 20 15

So, the starting block is 2015 in hex, which is, 8213 in decimal. Now, let us write a program for reading this sector from the SD Card dump and display it.

The program language I am going to use is Java. This just the computer language I use every day in my work, so for this reason I am going to use it.

First we need to open the dump:

RandomAccessFile fis = RandomAccessFile("dump.sdcard","r");

With instances of RandomAccessFile, we can easily jump around within different positions in the file, which is what will need for this exercise of attempting to read a file from a FAT32 partition in a dump file.

Now, using instances of RandomAccessFile, as well as any other classes which allows you to read from a file in Java, to surround it with a try-catch block to handle IOExceptions. However, to keep the conversation focussed, I will not go into details of exception handling in Java.

Next, let us add some code for seeking to the boot sector and reading it:

        fis.seek(8213 * 512);
        byte[] buf = new byte[512];
        fis.read(buf);
        var dataString = new String(buf, StandardCharsets.US_ASCII);
        System.out.println(dataString);

Obviously we need to multiply 8213 by 512, because the seek method wants the position in bytes. I then reaad the sector into a byte buffer and then convert it to a String to see if there is any interesting human readable properties. When printing the String, we see the following:

Interesting snippets, we see like MSDOS5.0 and BLACKBERRY FAT32. This is more or less strings we expect from a Bootsector formatted with FAT32. 

Some of you might be confused with the word BLACBERRY in the output. Well, I used a Blackberry Phone about 10 years ago. When I received the phone it became bundled with an SD Card, which I am using for this exercise.

This confirms that we found the correct sector as the boot sector. In the next section we go further looking into the info stored in this sector.

A deeper look into the Boot sector

To make sense of the info stored in the boot sector, the following resources is of great help:

https://wiki.osdev.org/FAT

As you can see, there is quite a number of sectors stored in Boot sector. We will only be needing a handful of these:

        sectorsPerFat = FatBrowser.readFourBytes(buf, 36);
        numFat = buf[16];
        numReserved = buf[14];
        sectorsPerCluster = buf[13];
        rootCluster = buf[44];
        dataStart = numReserved + numFat * sectorsPerFat;

Let us dissect this snippet of code a bit. buf is the byte buffer we read in the previous section, containing the boot sector. FatBrowser.readFourBytes() is a pseudo function for taking four bytes starting at position 36 of buf, and forming a number.

Now, with FAT32, which we use here, the available storage is divided into multiple clusters. Usually each cluster is more than one sector in size. In fact, with the SD Card I use in this post, each cluster is 64 sectors in size, indicated by the variable sectorsPerCluster.

With a cluster size of 64 sectors, it means that the size allocated per file will be at least 64 sectors in size and multiples of it. The FAT keeps track where the different parts of a file are on a disk and also works with clusters.

Having talked a lot about FATs and Clusters, let us see how these are arranged on the partition:

For the numbers in the diagram, I have used my SD Card as an example. The number in your case might differ.

In my case the partition starts with 9 reserved sector, of which the boot record is the first reserved sector.

Following the Reserved sectors comes the actual FAT, with one or more copies. The total number of FATs is indicated by byte 16 of the Boot record, indicated by the variable name numFat in the code snippet above. The actual size of 945 sectors of the FAT I got from the variable sectorsPerFAT.

The clusters that the FAT refers to lives in the Data Area. In my case, the Data Area begins at sector 9 + 945 + 945 = 1899.

Now, one may be tempted to say the first cluster in the Data Area is cluster number zero. This is, however, not the case with FAT, where the first cluster is numbered 2. The reason for this is because in the FAT table entries zero and one are reserved. The significance of this 2 is that any cluster number you obtain from the system, you need to subtract two to get the real cluster slot number within the Data Area.

The rootCluster in the code snippet above is no exception to the rule. In my case rootCluster is 2, meaning you will find the root directory at Slot 0 in the Data Area, e.g. right at the beginning of the Data Area.

The root directory contains actual file entries which we are interested in, which we will try to read in the next section.

Looking into the root directory

Let us have look at the how the first sector of the root directory looks like. The following Java snippet will read this sector and display it:

        fis.seek((8213 + 9 + 945+ 945) * 512);
        byte[] buf = new byte[512];
        fis.read(buf);
        var dataString = new String(buf, StandardCharsets.US_ASCII);
        for (int i = 0; i < 16; i++) {
            System.out.println(dataString.substring(0, 32));
            dataString = dataString.substring(32);
        }

The number we use in the seek I used as derived from the previous section, with 8213 the start of my FAT32 partition.

From the link I presented earlier on from osdev.org, I know each file entry in the root directory is 32 bytes, so I only print 32 bytes per line. The output I got from this program, looks like this:

Each line starts with something that looks like a filename. Some lines have characters which are separated by some whitespace. These characters are in actual fact Unicode characters which have two bytes per character.

These filenames with Unicode characters are actually entries for long filenames. For what we want to do, it is best to ignore long file entries, and just focus on the non-Unicode lines, like RECORD~1, BLACKB~1, MUSIC and so on. I will illustrate in a moment how we can filter out the long file entries.

Reading a file

Being able to view root directory entries, let us see if we can the contents of file. To do this, we first need to find the file entry in the root directory.

To do this, let us start by writing a code snippet for listing the filenames in the root directory, excluding the long file entries. To determine if an entry is a long file entry, we need to look at byte 11 of the file entry. From the link from OSDev.org, the bits in the attribute byte has the following meaning:

• READ_ONLY=0x01
• HIDDEN=0x02
• SYSTEM=0x04 
• VOLUME_ID=0x08 
• DIRECTORY=0x10 
• ARCHIVE=0x20 
• LFN=READ_ONLY|HIDDEN|SYSTEM|VOLUME_ID

For our purposes, we want to skip entries where the attribute is 15, resulting in the following code snippet:

        for (int i = 0; i < 16; i++) {
            int beginFileEntry = i * 32;
            if (buf[beginFileEntry + 11] == 15) {
                continue;
            }
            System.out.println(new String(buf, beginFileEntry, 11));
        }

This result in the following output:

This looks a lot cleaner than our previous attempt. One that looks strange in this output, however, is the entry preceded by the question mark. This is a deleted file, which we should also remove from our result. To skip past deleted files, we can just continue the loop as well if the first character of a filename is 0xE5.

Also, all the file entries we see here are directory entries. We need to search some more blocks in the is root directory cluster, to find some useful files to look at. So, we add an outer loop to our existing loop for reading the next block to look at:

        for (int j = 0; j < 63; j++) {
            fis.read(buf);
            for (int i = 0; i < 16; i++) {
                int beginFileEntry = i * 32;
                if (buf[beginFileEntry + 11] == 15) {
                    continue;
                }
                if ((buf[beginFileEntry] & 0xff) == 0xE5) {
                    continue;
                }
                System.out.println(new String(buf, beginFileEntry, 11));
            }
        }

This time we are seeing some more interesting stuff:

Here we see a couple of individual files, like WMPINFO.XML, CONTACTS.VCF, TEST.TXT and so on. Remember, these filenames are in 8.3 format. First 8 characters are the filename, followed by an extension of 3 characters. If the filename is less than 8 characters, you will space padding between filename and extension in output.

From these file entries, let us see if we can output the content of TEST.TXT. We modify our loops, so it will output the file entry slot within the current sector we are busy with:

        int foundSlot = -1;
        outerLoop:
        for (int j = 0; j < 63; j++) {
            fis.read(buf);
            for (int i = 0; i < 16; i++) {
                int beginFileEntry = i * 32;
                if (buf[beginFileEntry + 11] == 15) {
                    continue;
                }
                if ((buf[beginFileEntry] & 0xff) == 0xE5) {
                    continue;
                }
                if (new String(buf, beginFileEntry, 11).equalsIgnoreCase("TEST    TXT")) {
                    foundSlot = i * 32;
                    break outerLoop;
                }
            }
        }

Now, from the resulting file entry, we know the following:

• bytes 20 + 21: High 16 bits of first cluster number for file
• bytes 26 + 27: low 16 bits of first cluster number for file

To calculate the first cluster number of the file, we need to do some bit manipulation:

        int cluster = (buf[foundSlot + 21] & 0xff) >> 24 | (buf[foundSlot + 20] & 0xff) >> 16
                | (buf[foundSlot + 27] & 0xff) >> 8 | (buf[foundSlot + 26] & 0xff) >> 0;

Finally, we read the first sector of the file in question as follows:

        fis.seek((8213 + 9 + 945+ 945 + (cluster - 2) * 64) * 512);
        fis.read(buf);
        System.out.println(new String(buf));

In the seek command we basically start off again with calculation to find beginning of Data area, and adding to it the cluster number converted to a sector count. This is the output I get:

Ok, I admit I created this file beforehand and copied it to the SD Card, before making an image of it, which I used in this post😁

With all this done, I think we covered the basics of locating and reading a file from a FAT32 partition.

In Summary

In this post we continued our journey to find out how to read a file from an SD Card. We wrote some snippets of code to gradually explore how a FAT32 partition works and ended off by successfully reading file from such a partition.

In the next post we will redo the exercise, but by writing the code in 6502 Assembly language.

Until next time!

SD Card Access for a Arty A7: Part 6

Foreword

In the previous post we managed to Initialise an SD Card from power up.

In this post we will try to read a sector from the SD Card.

We will continue to use the Gisselquist SD Card core for interfacing with the SD Card. 

Buffering a sector of data

When you read a sector of data from a SD Card, the SD Card will respond after a certain amount of time at which it will send the 512 bytes of data one after the other consecutively. If your CPU is busy during this point in time, we might miss a byte or two from the data.

Luckily the Gisselquist core provide us way out of this scenario by buffering 512 bytes of data for us when it becomes available. The CPU can then fetch the data at a later stage from the buffer when it is ready.

This buffer is in actual fact a FIFO (First In First Out) structure. This means that you only need a single address to map the contents of the FIFO into CPU memory space and not 512 addresses.

There is a couple of technicalities to remember when using the FIFO buffers in the Gisselquist core. The first thing is, when you issue a read command to the SDCard, you should also inform the Gisselquist core that this command will be utilising the FIFO buffer. To illustrate this in 6502 assembly language syntax:

...
     .BYTE $00, $00, $00, $00
     .BYTE $00, $00, $08, $51 ; CMD 51
...

The value $51 is the SD Card command for reading a sector.

Next to the value $51 we have the value $08. If you study the final assembly listing from the previous post, you will pick up that usually for a command row the first three bytes of 0's, followed by a command byte. If there is any bits set in the first three bytes, it provides the Gisselquist core with some additional info about the particular command.

In this case the $08 byte signal the Gisselquist core that we expect 512 bytes from the SD Card and this should be stored in the FIFO buffer for later access.

Like with the other commands we need to wait in a busy wait loop until the busy bit changes to zero.

The next question is, how do we read the FIFO buffer? The answer is to read register 2 of the Gisselquist core. Let us recap from previous posts the registers that the Gisselquist core contains. I have added register 2 to the list, just for completeness:

• Register 0: Command register
• Register 1: Data register
• Register 2: FIFO buffer

These registers map into our 6502 address space starting at address FE00. It should be remembered that the Gisselquist core has 32 bit registers, whereas the 6502 works only with 8 bits at a time. So, Register 0 will map to address FE00, Register 1 will map to address FE04 and Register 2 will map to address FE08.

Now, back to the details of Register 2, the FIFO buffer register. In order to read the data you can continuously read register 2, which will bring you back 4 bytes at a time, with each read advancing to the next 4 bytes.

You will remember from my previous posts that in our 6502 design we trigger a register read by issuing an address that is a multiple of 4, like FE00, FE04 or FE08. The addresses FE01, FE02 and FE03 stores the remaining three bytes of the register which we couldn't accept, because the 6502 only works with a byte at a time.

There is one final technicality we need to look at. From the previous posts you will remember that we always use command $C0 for initialising the state of the Gisselquist core:

     .BYTE $55, $55, $55, $0B
     .BYTE $00, $00, $00, $C0 ; CMD C0

The value $0B set the value of the clock divider.

Now, the technicality I am referring to are bits 15-18, which specifies the limit of the FIFO buffer as a power of two. IF we use the command as above, the FIFO size limit value will be 5, which equals 32 bytes. The correct value to use for this is 9, which will yield the following $C0 command:

 

     .BYTE $55, $59, $55, $0B
     .BYTE $00, $00, $00, $C0 ; CMD C0

Verifying our design

When we run our design on a real FPGA, we will need a way to tell that the values are correctly read from the SD Card.

The easiest way for this verification is just to make an image dump of the SD Card we are using, and check some values with a Hex editor. The values we then get back from the FPGA need to match the values we saw in the Hex editor.

To take the SD Card I used as an example:

When I opened my dump, the beginning was filled with zeros. Not very useful for a test. However, Scrolling further down, eventually yielded some data:

So, if we run a test on the Arty A7, we should look at around byte 0x1bf of the sector data returned to verify if our implementation works correctly.

If you are doing a test yourself, you can also expect data in more or less the same spot. Any properly formatted SD Card will have a master boot record (MBR) located in sector zero. According to Wikipedia, bytes 0x1be to 0x1ce contains the first partition entry in a MBR.

Let us write some Assembly for reading sector zero from the SD Card:

       LDA #6
       JSR CMD
       LDA #0
       STA $FE0B
       LDX #$74
LOOPLD
       LDA $FE08
       DEX
       BNE LOOPLD
       LDA #2
       STA $FE0B

We start by issuing a sector read command. In our CMD table this is contained in slot 6, which we load into the Accumulator and we Jump to the CMD routine, which issues the command to the SD Card.

This routine only returns once we received the full response. Once the full response is loaded into the FIFO, our CPU needs to read it out, by reading address $FE08 multiple times.

We will again use an ILA (Integrated Logic Analyzer) block for examining the read data returned by the Gisselquist core to the CPU.

Unfortunately the data we are looking for is quite deep in the FIFO, and the Arty A7 doesn't provide enough block RAM to capture all read data returned. So, we need some trigger that we can only capture the segment we are interested in.

For this reason why are executing the loop in the above assembly $74 times. It should be noted that each read returns 4 bytes, so we need to multiply this number by 4 to get the real byte number where the loop will stop. In this case the byte number is $1d0, which gives us a big enough window for our ILA to capture all bytes in question.

Once the loop has completed, we store the value 2 into address $FE0B. This will set a bit triggering a capture on the ILA block. 

Let us have a look at what the ILA capture looks like:

As with a lot of ILA captures, they are simply too wide to present within a blog. With this ILA screen capture I tried to illustrate that I cut out a portion of the screenshot so we can view the important parts together. In the first part of signal we can see the assertion of the capture signal by the CPU at sector sample 0x1d0.

In the second signal we can see that the value 76000000. This is the value captured before we get to sample 0x1d0 and corresponds to the hex dump I presented earlier.

This proves that our design is more or less correct.

In Summary

In this post we attempted to read a sector of data from an SD Card and proved via a dump made from the SD Card via another system that our design read the data correctly.

In the next post we will attempt to read a file from an SD Card using its FAT table. This will be a big milestone in getting an Amiga core to boot up on an Arty A7, which will require to load an Amiga ROM and disk images from an SD Card.

Until next time!