I’ve recently got back from a trip and like everyone, I have a lot of pictures. I thought to myself that, it would be nice to arrange the pictures by date. Why not try and write some code to do that? Should be easy.
Let’s put the R in R&D
Quick google search taught me that this metadata I was after is called “Exif” data.
Why “Exif” you wonder? Well, I did of course, and it stands for “Exchangeable image file format”.
Alright, so a quick apt-get install got me a command line exif util. NOICE.
sudo apt-get install exif
This is what I got by feeding this cli tool a random pic I had on my disk.
EXIF tags in '~/Pictures/11.10.17/20120349_235641.jpg' ('Intel' byte order):
--------------------+----------------------------------------------------------
Tag |Value
--------------------+----------------------------------------------------------
Image Width |3264
Image Length |2448
Manufacturer |samsung
Model |SM-G5700
Orientation |Bottom-right
X-Resolution |72
Y-Resolution |72
Resolution Unit |Inch
Software |G5700ZHU1APL1
Date and Time |2017:09:29 23:33:42
YCbCr Positioning |Centered
Exposure Time |1/8 sec.
F-Number |f/1.9
Exposure Program |Normal program
ISO Speed Ratings |1000
Exif Version |Exif Version 2.2
Date and Time (Origi|2018:01:22 03:42:42
Date and Time (Digit|2018:01:22 03:42:42
Maximum Aperture Val|1.85 EV (f/1.9)
Metering Mode |Center-weighted average
Flash |Flash did not fire
Focal Length |2.5 mm
Maker Note |98 bytes undefined data
Color Space |sRGB
Pixel X Dimension |3264
Pixel Y Dimension |2448
Exposure Mode |Auto exposure
White Balance |Auto white balance
Focal Length in 35mm|24
Scene Capture Type |Standard
FlashPixVersion |FlashPix Version 1.0
--------------------+----------------------------------------------------------
Super cool!
So I can basically arrange my pictures by date, time, camera, size and all sorts of stuff.
Next I need to see what libraries I can use in order to extract that precious metadata.
I chose Go as my weapon of choice for that expedition, since I’ve been using it quite a lot lately and I can actually remember a lot of syntax.
Another google search showed this github library with a native go implementation for reading Exif data.
Hey, let’s wait a minute.. Do I really need this whole library just to read that “Exif” format…?
Well, not having to answer to anyone why I should do it myself and not “reinvent the wheel” is quite refreshing! I’ll arrange my photos later… That github repo is probably good for reference plus whatever I’ll find online. Ready? Set! Go!
I’m not running anywhere or trying to do this really fast
Finding where it starts
According to this document we need to find the APP1 marker. After the APP1 marker there will be the “APP1 data area” that I am looking for. Alright, so in this document it’s written that ‘The marker 0xFFE0~0xFFEF is named “Application Marker”’. What do I need to look for?
I’ve set up my code like this so I could go byte by byte and search for this “APP1” marker. BTW, I took some inspiration from the goexif library. The peek and discard technique is pretty nice.
package main
import (
"bufio"
"fmt"
"os"
)
func compBytes(a []byte, b []byte) bool {
lena := len(a)
if lena != len(b) { // if they are not equal in length they cannot be equal
return false
}
// check if any byte is different
for i := 0; i < lena; i++ {
if a[i] != b[i] {
return false
}
}
return true
}
func main () {
fname := "~/Pictures/11.10.17/20120349_235641.jpg"
fd, err := os.Open(fname) // open file read only
if err != nil {
fmt.Println("Error: ", err)
os.Exit(1)
}
defer fd.Close()
marker := []byte{0xF, 0xF, 0xE, 0x1}
br := bufio.NewReader(fd)
start := 0
bytesRead := 0
for {
bts, err := br.Peek(len(marker))
if err != nil {
fmt.Println(err)
break
}
if compBytes(bts, marker) {
fmt.Printf("found %s at %d\n", bts, start)
break
}
start++
bytesRead++
_, err = br.Discard(1)
}
fmt.Println("Bytes read: ", bytesRead)
fmt.Println("Start: ", start)
}
Well, that didn’t work… The file was read through to the end until getting EOF.
I guess I made some fundamental mistake because I cannot really grok this whole thing…
Maybe this is a byte containing a value and not a sequence of bytes?
...
func main () {
fname := "~/Pictures/11.10.17/20120349_235641.jpg"
fd, err := os.Open(fname) // open file read only
if err != nil {
fmt.Println("Error: ", err)
os.Exit(1)
}
defer fd.Close()
// marker := []byte{0xF, 0xF, 0xE, 0x1}
APP1 := 0xFFE1
br := bufio.NewReader(fd)
start := 0
bytesRead := 0
for {
// bts, err := br.Peek(len(marker))
bts, err := br.Peek(1)
if err != nil {
fmt.Println(err)
break
}
if bts[0] == byte(APP1) {
fmt.Println("eureka")
}
// if compBytes(bts, marker) {
// fmt.Printf("found %s at %d\n", bts, start)
// break
// }
start++
bytesRead++
_, err = br.Discard(1)
}
fmt.Println("Bytes read: ", bytesRead)
fmt.Println("Start: ", start)
}
Here’s what I got..
eureka
eureka
eureka
eureka
eureka
eureka
eureka
eureka
eureka
eureka
eureka
eureka
eureka
eureka
eureka
eureka
eureka
eureka
eureka
eureka
EOF
Bytes read: 2315185
Start: 2315185
Man… There are a lot of those. So maybe I’m right maybe I’m wrong. I mean.. Maybe there are a lot of those (?) The document says that after that header there’s ‘SSSS’ marking the size. Wish I knew what that means, pretty cryptic for all I know. Is it a byte array marking the size? is it just one byte as an int? Does S stands for size? OOooooofff… So much fun being a noob at something.
Ok, I’ll try printing the first few bytes. That might give me a hint.
ff
d8
ff
e1
Aha! It’s 2 bytes… OK, let’s keep going with this theory. Now my code looks like this:
...
func main () {
fname := "~/Pictures/11.10.17/20120349_235641.jpg"
fd, err := os.Open(fname) // open file read only
if err != nil {
fmt.Println("Error: ", err)
os.Exit(1)
}
defer fd.Close()
marker := []byte{0xFF, 0xE1}
// APP1 := 0xFFE1
br := bufio.NewReader(fd)
start := 0
bytesRead := 0
for {
bts, err := br.Peek(len(marker))
if err != nil {
fmt.Println(err)
break
}
if compBytes(bts, marker) {
fmt.Printf("found %x at %d\n", bts, start)
break
}
start++
bytesRead++
_, err = br.Discard(1)
}
fmt.Println("Bytes read: ", bytesRead)
fmt.Println("Start: ", start)
}
And returns this:
found ffe1 at 2
Bytes read: 2
Start: 2
Great! I can just found the FFE1 marker. Let’s see if I can find the following “Exif” string like the doc says: “Exif data is starts from ASCII character “Exif” and 2bytes of 0x00, and then Exif data follows.”
So this is where I’m at now, a bit ugly but we’ll make it nicer later.
....
for {
bts, err := br.Peek(len(marker))
if err != nil {
fmt.Println(err)
break
}
if compBytes(bts, marker) {
fmt.Printf("found %x at %d\n", bts, start)
br.Discard(2)
bts, _ := br.Peek(2)
// should be SSSS
vint := binary.BigEndian.Uint16(bts)
fmt.Println("vint: ", vint)
fmt.Printf("SSSS is %d\n", bts)
br.Discard(2)
bts, _ = br.Peek(4)
fmt.Println(fmt.Sprintf("Exif marker is %s", bts))
break
}
...
And here’s what I got:
found ffe1 at 2
vint: 638
SSSS is [2 126]
Exif marker is Exif
Extracting the data
Looks like we are on the right track! So, the size of our encoded data is 638 (determined by turning ‘SSSS’ to uint16). I’ll try to get that and turn it into a string.
...
bts, _ = br.Peek(4)
fmt.Println(fmt.Sprintf("Exif marker is %s", bts))
br.Discard(4)
bts, _ = br.Peek(2)
fmt.Println("should be 2 zeros", bts)
br.Discard(2)
exifData, _ := br.Peek(628) // minus 10 we already read
fmt.Printf("Exif data: %s\n", exifData)
...
We can already see there’s something there:
$ go run main.go
found ffe1 at 2
vint: 638
SSSS is [2 126]
Exif marker is Exif
should be 2 zeros [0 0]
Exif data: II*
�
�� ���(1�2�i��samsungSM-G5700G5700ZHU1APL12017:09:29 23:33:42HH��V��^"�'���0220�.�B�f� �
�n|�b����
�� ����0100
Z@P2017:09:29 23:33:422017:09:29 23:33:4�d�d�d
Bytes read: 2
Start: 2
When I look at it now I see that the data starts with II which according to the doc means it’s “Intel byte code”. I have been interpreting the size bytes as Big Endian but maybe it’s the other one.. So is our data 32238 or 638 bytes long?
Since the document doesn’t state which byte order to use in which occurrence I’ll go and dig into go-exif’s code and see how they solved it.
Found this bit:
BigEndianBoBytes = [2]byte{'M', 'M'}
LittleEndianBoBytes = [2]byte{'I', 'I'}
ByteOrderLookup = map[[2]byte]binary.ByteOrder{
BigEndianBoBytes: binary.BigEndian,
LittleEndianBoBytes: binary.LittleEndian,
}
ByteOrderLookupR = map[binary.ByteOrder][2]byte{
binary.BigEndian: BigEndianBoBytes,
binary.LittleEndian: LittleEndianBoBytes,
}
ExifFixedBytesLookup = map[binary.ByteOrder][2]byte{
binary.LittleEndian: {0x2a, 0x00},
binary.BigEndian: {0x00, 0x2a},
}
and this (inside a function):
byteOrderBytes := [2]byte{data[0], data[1]}
byteOrder, found := ByteOrderLookup[byteOrderBytes]
if found == false {
// exifLogger.Warningf(nil, "EXIF byte-order not recognized: [%v]", byteOrderBytes)
return eh, ErrNoExif
}
Looks like the first two bytes in a header determine the byte order. Cross referencing the two amazing resources I have (for real, not to be taken for granted) I understand that Intel code is little endian and Motorola is big.
The difference between big and small endian is the order the bytes are written at. Big endian is human readable - big numbers go last, but it depends where you start from…. Anyway if you want a detailed explanation, you can get one here
Before I continue I have to make the code a little bit more organized (this has turned into a mess).
Here I wrap the bufio.Reader I used before to give a more likable API and also added some nice utils:
exif/utils.go
package exif
import (
"bufio"
"fmt"
"io"
)
func compBytes(a []byte, b []byte) bool {
lena := len(a)
if lena != len(b) { // if they are not equal in length they cannot be equal
return false
}
// check if any byte is different
for i := 0; i < lena; i++ {
if a[i] != b[i] {
return false
}
}
return true
}
func forceASCII(s []byte) string {
rs := make([]byte, 0, len(s))
for _, r := range s {
if r <= 127 {
rs = append(rs, r)
}
}
return string(rs)
}
type WindowReader struct {
br *bufio.Reader
curr int
}
func (wr *WindowReader) NextWindow(winLength int) ([]byte, error) {
bts, err := wr.br.Peek(winLength)
if err != nil {
return nil, err
}
_, err = wr.br.Discard(winLength)
if err != nil {
return nil, err
}
wr.curr += winLength
return bts, nil
}
func (wr *WindowReader) ReadUntil(when int) ([]byte, error) {
readTo := make([]byte, when - wr.curr)
n, err := io.ReadFull(wr.br, readTo); if err != nil { return nil, err}
fmt.Println("Written: ", n)
wr.curr += n
return readTo, nil
}
func (wr *WindowReader) ScanUntil(toFind []byte) (bool, int, error) {
scanned := 0
for {
bts, err := wr.br.Peek(len(toFind))
if err != nil { return false, scanned, err } // Check for EOF?
if compBytes(bts, toFind) {
wr.br.Discard(len(toFind))
wr.curr += len(toFind)
return true, scanned, nil
}
_, err = wr.br.Discard(1)
if err != nil {
return false, scanned, err
}
scanned++
wr.curr++
}
}
func NewWindowReader(reader io.Reader) *WindowReader {
br := bufio.NewReader(reader)
return &WindowReader{br: br, curr: 0}
}
Copied everything from main to use my new fancy WindowReader:
exif/reader.go
package exif
import (
"encoding/binary"
"fmt"
"os"
)
const SizeDataLen = 2
const ExifStringDataLen = 4
var ExifMarker = []byte{0xFF, 0xE1}
func GetExifBytes(fname string) ([]byte, error) {
fd, err := os.Open(fname) // open file read only
if err != nil {
return nil, err
}
defer fd.Close()
wr := NewWindowReader(fd)
found, scanned, err := wr.ScanUntil(ExifMarker)
if err != nil { return nil, err }
if !found {
return nil, fmt.Errorf("could not read exif data from %s", fname)
}
sizeBytes, err := wr.NextWindow(SizeDataLen); if err != nil { return nil, err }
totalExifSize := binary.BigEndian.Uint16(sizeBytes)
fmt.Println("Exif size: ", totalExifSize)
exifStringBytes, err := wr.NextWindow(ExifStringDataLen); if err != nil { return nil, err }
exifString := forceASCII(exifStringBytes)
fmt.Println("Exif string: ", exifString)
if exifString != "Exif" {
return nil, fmt.Errorf("Unexpected non exif identifier")
}
twoZeros, err := wr.NextWindow(2); if err != nil { return nil, err } // should be two zeroes
if !compBytes(twoZeros, []byte{0, 0}) {
return nil, fmt.Errorf("Couldn't find the two zeros after the exif string")
}
fmt.Printf("Two Zeros: %x\n", twoZeros)
return nil, nil // TBD
}
Let’s try and adjust the code to handle different byte orders.
....
companyIDBts, err := wr.NextWindow(2); if err != nil { return nil, err } // should tell us if it's MM or II
fmt.Printf("CompanyID: %s\n", companyIDBts)
companyID := forceASCII(companyIDBts)
bom, err := wr.NextWindow(2); if err != nil { return nil, err } // either 0x2a00 or 0x002a
if companyID == "MM" && compBytes(bom, []byte{0x00, 0x2a}){
fileEndian = binary.BigEndian
} else if companyID == "II" && compBytes(bom, []byte{0x2a, 0x00}) {
fileEndian = binary.LittleEndian
} else {
return nil, fmt.Errorf("Could not figure out file byte order")
}
totalExifSize := fileEndian.Uint16(sizeBytes)
fmt.Println("Exif size: ", totalExifSize)
dataLenToRetrieve := int(totalExifSize) - wr.curr // total data includes the header
....
Alright! Markers, identifier, byte order.. All sorted! We now have a function that reads and returns the chunk of the file that holds our data.
Now we need to figure out how TIFF works. This document is a bit unclear about how exactly the data should be interpreted. There’s a table showing how the data is structured:
Next thing we have to do is to interpret the tiff header, which should give us the offset of IFD.
Figuring out IFD
After a long session of trial and error I finally understood what’s going on.
IFD or Image File Directory, holds the actual data (or keys and offsets to it)
Right after the Exif header there are 4 additional bytes that tell you the offset of the start of the IFD. That number is usually 8 since it starts after the Exif header.
Here’s our structure so far:
Jpeg marker and Exif header
| jpeg marker | Exif marker | Exif size | Two 0 | |
|---|---|---|---|---|
| Bytes | 2 | 2 | 4 | 2 |
| Value | 0xFFD8 (const) | 0xFFE1 (const) | Size of Exif data in bytes | 00 (const) |
Then the Tiff header
| Company ID | Byte Order (Endians) | Offset to first IFD | |
|---|---|---|---|
| Bytes | 2 | 2 | 4 |
| Value | Intel or Motorolla (MM/II) | 0x2a00 or in reverse | 32 bit int |
The offset to the first IFD as stated is probably 8.
The IFD
So here’s it gets a bit messy, but I’ll explain.
First let’s look at the IFD structure.
The first 2 bytes determine how many entries we have.
| Number of entries | |
|---|---|
| Bytes | 2 |
| Value | 16 bit uint |
After that each entry is divided like that:
| Tag ID | Data Format | Number of components/units | Offset/Data | |
|---|---|---|---|---|
| Bytes | 2 | 2 | 4 | 4 |
| Value | hex (to be matched in a table) | 16 bit int | 32 bit int | 32 bit int or *Other |
And now for the explanation.
The first 2 bytes of the IFD are representing the number of entries, so look at the entry table and imaging they appear one after the other without any breaks. For example, if ’number of entries’ is interpreted to 3 - there will be 3 entry “rows”. Each row will be consisted of 12 bytes as you can easily calculate.
The entry fields are used as following:
- Tag ID: Identifies the tag, there’s a table describing each tag and it’s description.
- 0x0110 for example is ‘Model’
- Data Format: Is an int between 1 and 12 determining the data type.
- For example: 2 is ascii, 12 is double float. There is a table for that too.
- Number Of components: The number of units of that data type.
- For example: we have an ascii entry (data type no. 2) and number of components is 4 (We have 4 ascii units).
- Since ascii has size of 2, means the size of our data is 8 bytes.
- Offset/Data: This field has two options
- When the data size in bytes is less than 4, it holds the data itself.
- When the data size in bytes is more than 4, it holds the offset to the data from the jpeg marker.
After the entries there are 4 zero bytes.
Now we can add code to our function to read all that information (we stopped at total exif size).
We should add a necessary translation table first.
type DataFormat struct {
Name string
BtsPerComponents uint32
}
var DataFormats = map[uint16]DataFormat{
1: {Name: "unsigned byte", BtsPerComponents: 1},
2: {Name: "ascii strings", BtsPerComponents: 1},
3: {Name: "unsigned short", BtsPerComponents: 2},
4: {Name: "unsigned long", BtsPerComponents: 4},
5: {Name: "unsigned rational", BtsPerComponents: 8},
6: {Name: "signed byte", BtsPerComponents: 1},
7: {Name: "undefined", BtsPerComponents: 1},
8: {Name: "signed short", BtsPerComponents: 2},
9: {Name: "signed long", BtsPerComponents: 4},
10: {Name: "signed rational", BtsPerComponents: 8},
11: {Name: "single float", BtsPerComponents: 4},
12: {Name: "double float", BtsPerComponents: 8},
}
Then we can continue with the IFD number of entries.
...
totalExifSize := fileEndian.Uint16(sizeBytes)
fmt.Println("Exif size: ", totalExifSize)
offsetToFirstBts, err := wr.NextWindow(4); if err != nil { return nil, err }
fmt.Println("offset to first ifd: ", fileEndian.Uint32(offsetToFirstBts)) // is 8
fmt.Println("Current offset: ", wr.curr)
// Start IFD
entryNumber, err := wr.NextWindow(2); if err != nil { return nil, err }
fmt.Println("Num of entries: ", fileEndian.Uint16(entryNumber))
numOfEntries := fileEndian.Uint16(entryNumber)
...
Now that we have the number of entries we can use a for loop to iterate them.
entryNumber, err := wr.NextWindow(2); if err != nil { return nil, err }
fmt.Println("Num of entries: ", fileEndian.Uint16(entryNumber))
numOfEntries := fileEndian.Uint16(entryNumber)
// For loop entries
for i := 0; i < int(numOfEntries) ; i++ {
// get Tag ID
tagID, err := wr.NextWindow(2) ; if err != nil { return nil, err }
fmt.Printf("tagID: %x\n", tagID)
// Data Format
dataFormatBts, err := wr.NextWindow(2) ; if err != nil { return nil, err }
dataFormat := fileEndian.Uint16(dataFormatBts)
fmt.Println("dataFormat: ", dataFormat)
// Num of components
NumOfComponentsBts, err := wr.NextWindow(4) ; if err != nil { return nil, err }
NumOfComponents := fileEndian.Uint32(NumOfComponentsBts)
fmt.Println("NumOfComponents: ", NumOfComponents)
// Offset/Data
offset, err := wr.NextWindow(4) ; if err != nil { return nil, err }
// use our data formats table
format := DataFormats[dataFormat]
fmt.Println("Format type is: ", format.Name)
// calculate the data size
nextDataSize := NumOfComponents * format.BtsPerComponents
if nextDataSize <= 4 { // in case the data is smaller than 4 bytes we can get it from here
switch dataFormat {
case 4:
fmt.Printf("Value is: %d\n\n", fileEndian.Uint16(offset))
case 3:
fmt.Printf("Value is: %d\n\n", fileEndian.Uint16(offset))
// and 10 more types....
}
continue // continue to the next entry
}
// read the data from the file descriptor
dataBts := make([]byte, nextDataSize)
fd.ReadAt(dataBts, int64(fileEndian.Uint32(offset) + uint32(2))) // the offset, as stated in the offset field
// plus 2 bytes for the jpeg marker
if format.Name == "ascii strings" {
fmt.Println("ascii: ", forceASCII(dataBts))
}
// and other types....
}
lastDirBytes, err := wr.NextWindow(4); if err != nil { return nil, err }
fmt.Printf("lastDirbytes %x\n", lastDirBytes) // should be 4 zeros
Alright!
The code is a bit ugly and only partially functional, but I bet you have a basic understanding of how this things work. I for sure had fun messing around with this encoded data. I just love to see a bunch of numbers and letters, that almost seem random, come together into logical data structures.
But don’t worry, there’s still work to be done! I would still like to have a nice and efficient implementation, that can read Exif data. Let’s see what I can come up with…
Reorganizing the code
Instead of having a big chunk of code, I decided to break the decoding into functions. Started with decoding the header:
exif/header.go
package exif
import (
"encoding/binary"
"fmt"
)
type CompanyID string
const Intel CompanyID = "II"
const Motorolla CompanyID = "MM"
const EXIF = "Exif"
var BigEndianOrder = []byte{0x00, 0x2a}
var LittleEndianOrder = []byte{0x2a, 0x00}
type ExifHeader struct {
Size uint16 // 2 b
ExifString string // 4 b 6
TwoZeros []byte // 2 b 8
CompanyID CompanyID // 2 b 10
ByteOrder binary.ByteOrder // 2 b 12
Offset uint32 // 4 b 16
}
// DecodeHeader accepts 16 bytes []byte and returns ExifHeader
func DecodeHeader(bts []byte) (ExifHeader, error) {
header := ExifHeader{}
if len(bts) != 16 {
return header, fmt.Errorf("expecting header length of 18 but got %d", len(bts))
}
sizeBts := bts[:2]
exifStrBts := bts[2:6]
twoZeros := bts[6:8]
companyIDBts := bts[8:10]
bomBts := bts[10:12]
offsetBts := bts[12:16]
var decodesResults []error
// Start with validations
err := validateExifString(exifStrBts); if err != nil { goto onError }
err = validateTwoZeros(twoZeros); if err != nil { goto onError }
// Continue on to determine byte order
header.CompanyID = CompanyID(forceASCII(companyIDBts))
header.ByteOrder, err = determineByteOrder(header.CompanyID, bomBts); if err != nil { goto onError }
// Now we can decode all those numbers
decodesResults = []error{
decode(header.ByteOrder, sizeBts, &header.Size),
decode(header.ByteOrder, offsetBts, &header.Offset),
}
for _, r := range decodesResults {
if r != nil {
err = r
goto onError
}
}
onError:
if err != nil{
return header, err
}
return header, nil
}
func determineByteOrder(cID CompanyID, bom []byte) (binary.ByteOrder, error) {
if cID == Motorolla && compBytes(bom, BigEndianOrder){
return binary.BigEndian, nil
} else if cID == Intel && compBytes(bom, LittleEndianOrder) {
return binary.LittleEndian, nil
}
return nil, fmt.Errorf("Could not figure out file byte order")
}
func validateExifString(bts []byte) error {
exifStr := forceASCII(bts)
if exifStr != EXIF {
return fmt.Errorf("Unexpected Exif identifier %s", exifStr)
}
return nil
}
func validateTwoZeros(bts []byte) error {
if !compBytes(bts, []byte{0, 0}) {
return fmt.Errorf("Expected 0x0000 but got %x", bts)
}
return nil
}
As you can probably see, I don’t really need all the fields in Header… But it helps me understand the byte structure so I keep it there.
This makes it possible to take the header from the file and decode it.
Note that I added a decode utility function (to utils.go), it looks like that:
func decode(bo binary.ByteOrder,b []byte, i interface{}) error {
switch i.(type) {
case *uint, *uint8, *uint16, *uint32, *uint64, *int, *int8, *int32, *int64, *float32, *float64:
return binary.Read(bytes.NewReader(b), bo, i)
case *string:
asc := forceASCII(b)
conv := i.(*string)
*conv = asc
default:
return fmt.Errorf("Unsupported type %T", i)
}
return nil
}
In my reader.go file I can now do:
....
wr := NewWindowReader(fd)
found, _, err := wr.ScanUntil(ExifMarker)
if err != nil { return nil, err }
if !found {
return nil, fmt.Errorf("could not read exif data from %s", fname)
}
headerBts, err := wr.NextWindow(16); if err != nil { return nil, err }
exifHeader, err := DecodeHeader(headerBts); if err != nil { return nil, err}
....
Already the code is more clear. Great!
Next, I added a decode for number of entries, even though it’s pretty small.
numentries.go
package exif
import "encoding/binary"
func DecodeNumEntries(byteOrder binary.ByteOrder, bts []byte) (int, error) {
var entries uint16
return int(entries), decode(byteOrder, bts, &entries)
}
And continue accordingly in our reader function:
....
fileEndian = exifHeader.ByteOrder
if int(exifHeader.Offset) > wr.curr {
wr.NextWindow(int(exifHeader.Offset) - wr.curr)
}
// Start IFD
entryNumber, err := wr.NextWindow(2); if err != nil { return nil, err }
numOfEntries, err := DecodeNumEntries(fileEndian, entryNumber); if err != nil { return nil, err }
....
After that was done, all that was left is to decode the entry.
entry.go
package exif
import (
"encoding/binary"
"fmt"
"io"
)
type Entry struct {
Tag string // 2
DataFormat uint16 // 2 4
NumOfComponents uint32 // 4 8
Offset uint32 // 4 12
offsetBts []byte
byteOrder binary.ByteOrder
}
type DataFormat struct {
Name string
BtsPerComponents uint32
ID uint16
}
func (ent Entry) GetDataFormat() DataFormat {
return dataFormats[ent.DataFormat]
}
func (ent Entry) DataSize() int {
df := ent.GetDataFormat()
return int(ent.NumOfComponents * df.BtsPerComponents)
}
func (ent Entry) DecodeData(rd io.ReaderAt, i interface{}) error {
dataSize := ent.DataSize()
if dataSize <= 4 { // in case the data is smaller than 4 bytes we can get it from the offset bytes
return decode(ent.byteOrder, ent.offsetBts, i)
}
dataBts := make([]byte, dataSize)
_, err := rd.ReadAt(dataBts, int64(ent.Offset) + 12 ) // the offset, as stated in the offset field
if err != nil { // plus 12 bytes for the header
return err
}
return decode(ent.byteOrder, dataBts, i)
}
const (
UByte uint16 = 1
AsciiString uint16 = 2
UShort uint16 = 3
ULong uint16 = 4
Urational uint16 = 5
Byte uint16 = 6
Undefined uint16 = 7
Short uint16 = 8
Long uint16 = 9
Rational uint16 = 10
Float uint16 = 11
DoubleFloat uint16 = 12
)
var dataFormats = map[uint16]DataFormat{
1: {ID: 1, Name: "unsigned byte", BtsPerComponents: 1},
2: {ID: 2, Name: "ascii strings", BtsPerComponents: 1},
3: {ID: 3, Name: "unsigned short", BtsPerComponents: 2},
4: {ID: 4, Name: "unsigned long", BtsPerComponents: 4},
5: {ID: 5, Name: "unsigned rational", BtsPerComponents: 8},
6: {ID: 6, Name: "signed byte", BtsPerComponents: 1},
7: {ID: 7, Name: "undefined", BtsPerComponents: 1},
8: {ID: 8, Name: "signed short", BtsPerComponents: 2},
9: {ID: 9, Name: "signed long", BtsPerComponents: 4},
10: {ID: 10, Name: "signed rational", BtsPerComponents: 8},
11: {ID: 11, Name: "single float", BtsPerComponents: 4},
12: {ID: 12, Name: "double float", BtsPerComponents: 8},
}
// DecodeEntry accepts a 12 byte array and returns an Entry
func DecodeEntry(bo binary.ByteOrder, bts []byte) (Entry, error) {
entry := Entry{
byteOrder: bo,
}
if len(bts) != 12 {
return entry, fmt.Errorf("Expected a 12 bytes long slice but got %d", len(bts))
}
if bo == binary.BigEndian {
entry.Tag = fmt.Sprintf("0x%x",string(bts[:2]))
} else {
entry.Tag = fmt.Sprintf("0x%x",string([]byte{bts[1], bts[0]})) // need to flip it
}
dataFormatBts := bts[2:4]
numOFComponentsBts := bts[4:8]
entry.offsetBts = bts[8:12]
errs := []error {
decode(bo, dataFormatBts, &entry.DataFormat),
decode(bo, numOFComponentsBts, &entry.NumOfComponents),
decode(bo, entry.offsetBts, &entry.Offset),
}
for _, err := range errs {
if err != nil {
return entry, err
}
}
return entry, nil
}
Wooow! That’s a lot of code… So, I started by decoding the entry just like I did with the header. After that I did some grunt work and added the data format list and consts. And finally I added the DecodeData function.
By the way I’m not very sure about this:
_, err := rd.ReadAt(dataBts, int64(ent.Offset) + 12 )
The header is actually 16 bytes. So the offset field is not very clear and the question to be asked is - “offset from where?”, because it’s not from the beginning of the file or from the beginning of exif data…
Anyway, at the moment I can live with 12 being the magic number.
I also did some more grunt work for the tags (copied some from the doc, of course not all of them)
tags.go
package exif
type Tag struct {
ID string
Name string
}
// names are in big endian
var Tags = map[string]Tag{
"0x0100": {ID: "0x0100", Name: "ImageWidth"}, // ImageWidth has double tag
"0x0001": {ID: "0x0001", Name: "ImageWidth"},
"0x0101": {ID: "0x0101", Name: "ImageLength"},
"0x9003": {ID: "0x9003", Name:"DateTimeOriginal"},
"0x0132": {ID: "0x0132", Name:"DateTime"},
"0x010f": {ID: "0x010f", Name: "Make"},
"0x0110": {ID: "0x0110", Name: "Model"},
"0x0112": {ID: "0x0112", Name: "Orientation"},
"0x0131": {ID: "0x0131", Name: "Software"},
}
And now the final version of my reader function looks like that:
reader.go
package exif
import (
"encoding/binary"
"fmt"
"os"
)
const SizeDataLen = 2
const ExifStringDataLen = 4
var ExifMarker = []byte{0xFF, 0xE1}
var fileEndian binary.ByteOrder
func GetExifData(fname string) (map[string]interface{}, error) {
res := map[string]interface{}{}
fd, err := os.Open(fname) // open file read only
if err != nil {
return nil, err
}
defer fd.Close()
wr := NewWindowReader(fd)
found, _, err := wr.ScanUntil(ExifMarker)
if err != nil { return nil, err }
if !found {
return nil, fmt.Errorf("could not read exif data from %s", fname)
}
headerBts, err := wr.NextWindow(16); if err != nil { return nil, err }
exifHeader, err := DecodeHeader(headerBts); if err != nil { return nil, err}
fileEndian = exifHeader.ByteOrder
if int(exifHeader.Offset) > wr.curr {
wr.NextWindow(int(exifHeader.Offset) - wr.curr)
}
// Start IFD
entryNumber, err := wr.NextWindow(2); if err != nil { return nil, err }
numOfEntries, err := DecodeNumEntries(fileEndian, entryNumber); if err != nil { return nil, err }
// For loop entries
for i := 0; i < numOfEntries ; i++ {
entryBts, err := wr.NextWindow(12); if err != nil { return nil, err}
entry, err := DecodeEntry(fileEndian, entryBts); if err != nil { return nil, err}
df := entry.GetDataFormat()
// uint16, *uint32, *int64, *int, *int32, *float32, *float64
tag, ok := Tags[entry.Tag]
if ok {
var d interface{}
switch df.ID {
case UByte:
var v uint8
err = entry.DecodeData(fd, &v)
d = v
case AsciiString:
var v string
err = entry.DecodeData(fd, &v)
d = v
case UShort:
var v uint16
err = entry.DecodeData(fd, &v)
d = v
case ULong:
var v uint32
err = entry.DecodeData(fd, &v)
d = v
case Urational:
var v uint64
err = entry.DecodeData(fd, &v)
d = v
case Byte:
var v int8
err = entry.DecodeData(fd, &v)
d = v
case Undefined:
// leave it interface
case Short:
var v int16
err = entry.DecodeData(fd, &v)
d = v
case Long:
var v int32
err = entry.DecodeData(fd, &v)
d = v
case Rational:
var v int64
err = entry.DecodeData(fd, &v)
d = v
case Float:
var v float32
err = entry.DecodeData(fd, &v)
d = v
case DoubleFloat:
var v float64
err = entry.DecodeData(fd, &v)
d = v
}
if err != nil { return nil, err}
res[tag.Name] = d
}
}
// lastDirBytes, err := wr.NextWindow(4); if err != nil { return nil, err }
// fmt.Printf("lastDirbytes %x\n", lastDirBytes) // should be 4 zeros
return res, nil
}
It’s fairly readable.. Would maybe move that ugly switch case, but I think we’re good for now.
Now my main function:
func main () {
fname := "~/Pictures/11.10.17/20120349_235641.jpg"
props, err := exif.GetExifData(fname)
if err != nil {
fmt.Println("Error: ", err)
return
}
for k, v := range props {
fmt.Println(k,": ", v)
}
}
And the output is something like that:
$ go run main.go
Software : J7800GUOPAPL1
DateTime : 2021:06:01 66:66:66
ImageWidth : 3264
ImageLength : 2448
Make : xiaomi
Model : UX-8866
That’s all for today! You can check out the complete solution at https://github.com/asafg6/exif-reader
Hope you enjoyed!
